==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-647987fcd9-b4ghf Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-frontend-18a44ad7-wxs9/10.142.0.11 Start Time: Fri, 20 Dec 2024 18:25:34 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=647987fcd9 skaffold.dev/run-id=5dad857a-13eb-4832-925e-8d1eccccab32 Annotations: Status: Running IP: 10.106.40.178 IPs: IP: 10.106.40.178 Controlled By: ReplicaSet/lodemon-647987fcd9 Containers: lodemon: Container ID: containerd://f6f759c89b41a63e3040ad89aadb7f449e0b3fe823f8bfd45816e63e670081c1 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:4ba4f6066d128cbe60604a46e6c908ecfbc82c97c0cb359b8c9fdd92c9ab55a1 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 20 Dec 2024 18:25:35 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: idc ds-only Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fnm66 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-fnm66: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== 09:43:51 INFO 09:43:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:43:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:43:51 INFO [loop_until]: OK (rc = 0) 09:43:51 DEBUG --- stdout --- 09:43:51 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13707Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:43:51 DEBUG --- stderr --- 09:43:51 DEBUG 09:43:56 INFO 09:43:56 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:43:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:43:56 INFO [loop_until]: OK (rc = 0) 09:43:56 DEBUG --- stdout --- 09:43:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 104m 0% 3176Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 103m 0% 19859Mi 69% 09:43:56 DEBUG --- stderr --- 09:43:56 DEBUG 09:44:08 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774248" 09:44:08 INFO [http_cmd]: http status code OK 09:44:08 DEBUG --- status code --- 09:44:08 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:08 DEBUG --- http response --- 09:44:08 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 ERROR [http_cmd]: ERROR 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 ERROR --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 ERROR --- http response --- 09:44:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:44:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:44:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG { 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "data": { 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "result": [ 09:44:09 DEBUG { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "pod": "ds-cts-0" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG [...] 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "558.5447742403362" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG { 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "data": { 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "result": [ 09:44:09 DEBUG { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "pod": "ds-cts-0" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG [...] 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "731250688" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774249" 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774249" 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774249" 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG { 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "data": { 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "result": [ 09:44:09 DEBUG { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "instance": "10.142.0.11:9100" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG [...] 09:44:09 DEBUG 09:44:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774249" 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG "instance": "10.142.0.51:9100" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "67555.50000000001" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG { 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "data": { 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "result": [ 09:44:09 DEBUG { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG [...] 09:44:09 DEBUG 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG --- status code --- 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG }, 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG "value": [ 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 INFO [http_cmd]: http status code OK 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 DEBUG { 09:44:09 DEBUG { 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG "0.03551287230303855" 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG --- status code --- 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "data": { 09:44:09 DEBUG { 09:44:09 DEBUG ] 09:44:09 DEBUG { 09:44:09 DEBUG { 09:44:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:44:09 DEBUG { 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG } 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG --- http response --- 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "data": { 09:44:09 DEBUG "result": [ 09:44:09 DEBUG "data": { 09:44:09 DEBUG ] 09:44:09 DEBUG "data": { 09:44:09 DEBUG "data": { 09:44:09 DEBUG "data": { 09:44:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG { 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG } 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG ----- output ----- 09:44:09 DEBUG { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "result": [ 09:44:09 DEBUG } 09:44:09 DEBUG "result": [ 09:44:09 DEBUG "result": [ 09:44:09 DEBUG "result": [ 09:44:09 DEBUG "result": [ 09:44:09 DEBUG "status": "success", 09:44:09 DEBUG "instance": "10.142.0.11:9100" 09:44:09 DEBUG { 09:44:09 DEBUG { 09:44:09 DEBUG { 09:44:09 DEBUG { 09:44:09 DEBUG { 09:44:09 DEBUG "data": { 09:44:09 DEBUG }, 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "metric": { 09:44:09 DEBUG "resultType": "vector", 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "pod": "ds-cts-0" 09:44:09 DEBUG "pod": "ds-cts-0" 09:44:09 DEBUG "instance": "10.142.0.11:9100" 09:44:09 DEBUG "instance": "10.142.0.11:9100" 09:44:09 DEBUG "instance": "10.142.0.11:9100" 09:44:09 DEBUG "result": [ 09:44:09 DEBUG 09:44:09 DEBUG }, 09:44:09 DEBUG }, 09:44:09 DEBUG }, 09:44:09 DEBUG }, 09:44:09 DEBUG }, 09:44:09 DEBUG { 09:44:09 DEBUG 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "metric": { 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:44:09 DEBUG [...] 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG }, 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 09:44:09 DEBUG [...] 09:44:09 DEBUG [...] 09:44:09 DEBUG [...] 09:44:09 DEBUG [...] 09:44:09 DEBUG [...] 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG "instance": "10.142.0.51:9100" 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG }, 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG 09:44:09 DEBUG [...] 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG "instance": "10.142.0.51:9100" 09:44:09 DEBUG "instance": "10.142.0.51:9100" 09:44:09 DEBUG 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "instance": "10.142.0.51:9100" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 09:44:09 DEBUG "85889.29999999999" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG 09:44:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:44:09 DEBUG }, 09:44:09 DEBUG "value": [ 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "15775481856" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG "0" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG }, 09:44:09 DEBUG "26760.533333333333" 09:44:09 DEBUG }, 09:44:09 DEBUG "0" 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG "value": [ 09:44:09 DEBUG ] 09:44:09 DEBUG "value": [ 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG } 09:44:09 DEBUG 1734774249, 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG "0.0005333333333349326" 09:44:09 DEBUG ] 09:44:09 DEBUG "0.0011507058199259374" 09:44:09 DEBUG ] 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG ] 09:44:09 DEBUG ] 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:09 DEBUG } 09:44:51 INFO 09:44:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:44:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:44:51 INFO [loop_until]: OK (rc = 0) 09:44:51 DEBUG --- stdout --- 09:44:51 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:44:51 DEBUG --- stderr --- 09:44:51 DEBUG 09:44:56 INFO 09:44:56 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:44:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:44:56 INFO [loop_until]: OK (rc = 0) 09:44:56 DEBUG --- stdout --- 09:44:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 103m 0% 3169Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 19860Mi 69% 09:44:56 DEBUG --- stderr --- 09:44:56 DEBUG 09:45:08 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774308" 09:45:08 INFO [http_cmd]: http status code OK 09:45:08 DEBUG --- status code --- 09:45:08 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:08 DEBUG --- http response --- 09:45:08 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- http response --- 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- http response --- 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 ERROR [http_cmd]: ERROR 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- status code --- 09:45:09 ERROR --- status code --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG --- http response --- 09:45:09 ERROR --- http response --- 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:45:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:45:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG { 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "result": [ 09:45:09 DEBUG { 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "pod": "ds-idrepo-0" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "pod": "ds-cts-0" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG "476.90237038232846" 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG { 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "result": [ 09:45:09 DEBUG { 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "pod": "ds-cts-0" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG "731267072" 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG { 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "result": [ 09:45:09 DEBUG { 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "instance": "10.142.0.11:9100" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774309" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "instance": "10.142.0.51:9100" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG "67648.16666666667" 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG { 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "result": [ 09:45:09 DEBUG { 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG "0.0444116942856419" 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774309" 09:45:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774309" 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG "data": { 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 DEBUG { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG { 09:45:09 DEBUG { 09:45:09 DEBUG "result": [ 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG "result": [ 09:45:09 DEBUG "data": { 09:45:09 DEBUG { 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "metric": { 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG "result": [ 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "result": [ 09:45:09 DEBUG "instance": "10.142.0.11:9100" 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG { 09:45:09 DEBUG "instance": "10.142.0.11:9100" 09:45:09 DEBUG { 09:45:09 DEBUG }, 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG "metric": { 09:45:09 DEBUG }, 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "value": [ 09:45:09 INFO [http_cmd]: http status code OK 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:45:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:45:09 DEBUG "value": [ 09:45:09 DEBUG "instance": "10.142.0.11:9100" 09:45:09 DEBUG 09:45:09 DEBUG --- status code --- 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG }, 09:45:09 DEBUG 09:45:09 DEBUG }, 09:45:09 DEBUG 09:45:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG { 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG --- http response --- 09:45:09 DEBUG { 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "data": { 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG ----- output ----- 09:45:09 DEBUG "data": { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG { 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "result": [ 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "status": "success", 09:45:09 DEBUG "result": [ 09:45:09 DEBUG { 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "data": { 09:45:09 DEBUG { 09:45:09 DEBUG "metric": { 09:45:09 DEBUG 09:45:09 DEBUG "instance": "10.142.0.51:9100" 09:45:09 DEBUG "instance": "10.142.0.51:9100" 09:45:09 DEBUG 09:45:09 DEBUG "resultType": "vector", 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "pod": "ds-idrepo-0" 09:45:09 DEBUG 09:45:09 DEBUG }, 09:45:09 DEBUG }, 09:45:09 DEBUG "instance": "10.142.0.51:9100" 09:45:09 DEBUG "result": [ 09:45:09 DEBUG "pod": "ds-idrepo-0" 09:45:09 DEBUG }, 09:45:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:45:09 DEBUG "value": [ 09:45:09 DEBUG "value": [ 09:45:09 DEBUG }, 09:45:09 DEBUG { 09:45:09 DEBUG }, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG }, 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG "value": [ 09:45:09 DEBUG "metric": { 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG "value": [ 09:45:09 DEBUG "33450.666666666664" 09:45:09 DEBUG "85001.73333333331" 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG "instance": "10.142.0.11:9100" 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG 09:45:09 DEBUG "15778643968" 09:45:09 DEBUG ] 09:45:09 DEBUG "0.00033333333333350387" 09:45:09 DEBUG }, 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG 09:45:09 DEBUG 09:45:09 DEBUG "pod": "ds-cts-0" 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG [...] 09:45:09 DEBUG 09:45:09 DEBUG }, 09:45:09 DEBUG 09:45:09 DEBUG "pod": "ds-cts-0" 09:45:09 DEBUG "value": [ 09:45:09 DEBUG 09:45:09 DEBUG }, 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG 09:45:09 DEBUG "value": [ 09:45:09 DEBUG "0" 09:45:09 DEBUG "instance": "10.142.0.51:9100" 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG ] 09:45:09 DEBUG }, 09:45:09 DEBUG "0.0025899547797776827" 09:45:09 DEBUG } 09:45:09 DEBUG "value": [ 09:45:09 DEBUG ] 09:45:09 DEBUG ] 09:45:09 DEBUG 1734774309, 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG "0.0005333333333349326" 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:09 DEBUG ] 09:45:09 DEBUG } 09:45:09 DEBUG } 09:45:51 INFO 09:45:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:45:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:45:51 INFO [loop_until]: OK (rc = 0) 09:45:51 DEBUG --- stdout --- 09:45:51 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:45:51 DEBUG --- stderr --- 09:45:51 DEBUG 09:45:56 INFO 09:45:56 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:45:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:45:56 INFO [loop_until]: OK (rc = 0) 09:45:56 DEBUG --- stdout --- 09:45:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 101m 0% 3170Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 94m 0% 19860Mi 69% 09:45:56 DEBUG --- stderr --- 09:45:56 DEBUG 09:46:08 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774368" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG --- status code --- 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- status code --- 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 ERROR [http_cmd]: ERROR 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 ERROR --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 ERROR --- http response --- 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:46:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG { 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "data": { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "result": [ 09:46:09 DEBUG { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "pod": "ds-cts-0" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "pod": "ds-idrepo-0" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "716.8860263231588" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG { 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "data": { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "result": [ 09:46:09 DEBUG { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "pod": "ds-cts-0" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "731275264" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG { 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "data": { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "result": [ 09:46:09 DEBUG { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "instance": "10.142.0.11:9100" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "instance": "10.142.0.51:9100" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "70606.59999999999" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG { 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "data": { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "result": [ 09:46:09 DEBUG { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "0.050827206593793334" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774369" 09:46:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774369" 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG { 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "data": { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "result": [ 09:46:09 DEBUG { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "instance": "10.142.0.11:9100" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "instance": "10.142.0.51:9100" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "25531.733333333334" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG { 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "data": { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "result": [ 09:46:09 DEBUG { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "15776247808" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG { 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG "data": { 09:46:09 DEBUG { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "status": "success", 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG "result": [ 09:46:09 DEBUG "data": { 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG { 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "result": [ 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG "instance": "10.142.0.11:9100" 09:46:09 DEBUG { 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:46:09 DEBUG }, 09:46:09 DEBUG "metric": { 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG "value": [ 09:46:09 DEBUG "instance": "10.142.0.11:9100" 09:46:09 DEBUG { 09:46:09 DEBUG { 09:46:09 DEBUG 09:46:09 DEBUG }, 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG 09:46:09 DEBUG "value": [ 09:46:09 DEBUG "data": { 09:46:09 DEBUG "data": { 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG [...] 09:46:09 DEBUG 09:46:09 DEBUG "result": [ 09:46:09 DEBUG "result": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG { 09:46:09 DEBUG { 09:46:09 DEBUG 09:46:09 DEBUG [...] 09:46:09 INFO [http_cmd]: http status code OK 09:46:09 DEBUG "metric": { 09:46:09 DEBUG "metric": { 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG --- status code --- 09:46:09 DEBUG "pod": "ds-cts-0" 09:46:09 DEBUG "instance": "10.142.0.11:9100" 09:46:09 DEBUG "instance": "10.142.0.51:9100" 09:46:09 DEBUG 09:46:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:46:09 DEBUG }, 09:46:09 DEBUG }, 09:46:09 DEBUG }, 09:46:09 DEBUG 09:46:09 DEBUG --- http response --- 09:46:09 DEBUG "value": [ 09:46:09 DEBUG "value": [ 09:46:09 DEBUG "value": [ 09:46:09 DEBUG "instance": "10.142.0.51:9100" 09:46:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG }, 09:46:09 DEBUG ----- output ----- 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG "0.0003333333333330302" 09:46:09 DEBUG "value": [ 09:46:09 DEBUG { 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG ] 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "status": "success", 09:46:09 DEBUG [...] 09:46:09 DEBUG [...] 09:46:09 DEBUG } 09:46:09 DEBUG "88147.56666666667" 09:46:09 DEBUG "data": { 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG ] 09:46:09 DEBUG ] 09:46:09 DEBUG "resultType": "vector", 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 DEBUG "result": [ 09:46:09 DEBUG 09:46:09 DEBUG 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG { 09:46:09 DEBUG "pod": "ds-idrepo-0" 09:46:09 DEBUG "instance": "10.142.0.51:9100" 09:46:09 DEBUG } 09:46:09 DEBUG "metric": { 09:46:09 DEBUG }, 09:46:09 DEBUG }, 09:46:09 DEBUG } 09:46:09 DEBUG "pod": "ds-cts-0" 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG "0.005426036124169833" 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 09:46:09 DEBUG ] 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG 09:46:09 DEBUG } 09:46:09 DEBUG "0.00043333333333350765" 09:46:09 DEBUG 09:46:09 DEBUG ] 09:46:09 DEBUG ] 09:46:09 DEBUG [...] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:09 DEBUG 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG 09:46:09 DEBUG } 09:46:09 DEBUG 09:46:09 DEBUG } 09:46:09 DEBUG "pod": "ds-idrepo-0" 09:46:09 DEBUG }, 09:46:09 DEBUG "value": [ 09:46:09 DEBUG 1734774369, 09:46:09 DEBUG "0" 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG ] 09:46:09 DEBUG } 09:46:09 DEBUG } 09:46:51 INFO 09:46:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:46:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:46:51 INFO [loop_until]: OK (rc = 0) 09:46:51 DEBUG --- stdout --- 09:46:51 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:46:51 DEBUG --- stderr --- 09:46:51 DEBUG 09:46:56 INFO 09:46:56 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:46:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:46:57 INFO [loop_until]: OK (rc = 0) 09:46:57 DEBUG --- stdout --- 09:46:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3174Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 97m 0% 19860Mi 69% 09:46:57 DEBUG --- stderr --- 09:46:57 DEBUG 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 ERROR [http_cmd]: ERROR 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 ERROR --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 ERROR --- http response --- 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:47:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:47:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "data": { 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "pod": "ds-cts-0" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG [...] 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "821.1098292622135" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "data": { 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "pod": "ds-cts-0" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG [...] 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "731480064" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "data": { 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "instance": "10.142.0.11:9100" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG [...] 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "instance": "10.142.0.51:9100" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "67650" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "data": { 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG [...] 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "0.04199969247031708" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "data": { 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "instance": "10.142.0.11:9100" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG [...] 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "instance": "10.142.0.51:9100" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "23074.13333333333" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774429" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774429" 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 DEBUG "data": { 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774429" 09:47:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774429" 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG [...] 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "15778742272" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "data": { 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG "metric": { 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG "instance": "10.142.0.11:9100" 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG }, 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG "value": [ 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG ----- output ----- 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG 09:47:09 DEBUG { 09:47:09 DEBUG { 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG [...] 09:47:09 DEBUG "data": { 09:47:09 DEBUG "data": { 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:47:09 DEBUG 09:47:09 DEBUG "result": [ 09:47:09 DEBUG "result": [ 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG 09:47:09 DEBUG { 09:47:09 DEBUG { 09:47:09 DEBUG { 09:47:09 DEBUG "instance": "10.142.0.51:9100" 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG }, 09:47:09 DEBUG "pod": "ds-cts-0" 09:47:09 DEBUG "instance": "10.142.0.11:9100" 09:47:09 DEBUG "data": { 09:47:09 DEBUG "value": [ 09:47:09 DEBUG }, 09:47:09 DEBUG }, 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG "value": [ 09:47:09 DEBUG "result": [ 09:47:09 DEBUG { 09:47:09 DEBUG "metric": { 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "0.000666666666666534" 09:47:09 INFO [http_cmd]: http status code OK 09:47:09 DEBUG "instance": "10.142.0.11:9100" 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG ] 09:47:09 DEBUG --- status code --- 09:47:09 DEBUG }, 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG } 09:47:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:47:09 DEBUG "value": [ 09:47:09 DEBUG [...] 09:47:09 DEBUG [...] 09:47:09 DEBUG ] 09:47:09 DEBUG --- http response --- 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG } 09:47:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG } 09:47:09 DEBUG ----- output ----- 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG { 09:47:09 DEBUG [...] 09:47:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:47:09 DEBUG "instance": "10.142.0.51:9100" 09:47:09 DEBUG "status": "success", 09:47:09 DEBUG 09:47:09 DEBUG }, 09:47:09 DEBUG }, 09:47:09 DEBUG "data": { 09:47:09 DEBUG 09:47:09 DEBUG "value": [ 09:47:09 DEBUG "value": [ 09:47:09 DEBUG "resultType": "vector", 09:47:09 DEBUG 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "result": [ 09:47:09 DEBUG "instance": "10.142.0.51:9100" 09:47:09 DEBUG "0.0015507897909294611" 09:47:09 DEBUG "85077.29999999999" 09:47:09 DEBUG { 09:47:09 DEBUG }, 09:47:09 DEBUG ] 09:47:09 DEBUG ] 09:47:09 DEBUG "metric": { 09:47:09 DEBUG "value": [ 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 DEBUG "pod": "ds-cts-0" 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG ] 09:47:09 DEBUG ] 09:47:09 DEBUG }, 09:47:09 DEBUG "0.0010666666666660754" 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 DEBUG "value": [ 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:09 DEBUG 09:47:09 DEBUG } 09:47:09 DEBUG 09:47:09 DEBUG ] 09:47:09 DEBUG 09:47:09 DEBUG } 09:47:09 DEBUG [...] 09:47:09 DEBUG } 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG 09:47:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:47:09 DEBUG }, 09:47:09 DEBUG "value": [ 09:47:09 DEBUG 1734774429, 09:47:09 DEBUG "0" 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG ] 09:47:09 DEBUG } 09:47:09 DEBUG } 09:47:51 INFO 09:47:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:47:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:47:51 INFO [loop_until]: OK (rc = 0) 09:47:51 DEBUG --- stdout --- 09:47:51 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:47:51 DEBUG --- stderr --- 09:47:51 DEBUG 09:47:57 INFO 09:47:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:47:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:47:57 INFO [loop_until]: OK (rc = 0) 09:47:57 DEBUG --- stdout --- 09:47:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3175Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 98m 0% 19854Mi 69% 09:47:57 DEBUG --- stderr --- 09:47:57 DEBUG 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG --- status code --- 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG --- status code --- 09:48:09 ERROR [http_cmd]: ERROR 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 ERROR --- status code --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG --- http response --- 09:48:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 ERROR --- http response --- 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:48:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:48:09 DEBUG ----- output ----- 09:48:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "pod": "ds-cts-0" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "507.07100432210007" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "pod": "ds-cts-0" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "731299840" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "instance": "10.142.0.11:9100" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "instance": "10.142.0.51:9100" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "69582.7" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "0.04042628681139762" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "instance": "10.142.0.11:9100" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "instance": "10.142.0.51:9100" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "25804.8" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774489" 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "15775281152" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774489" 09:48:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774489" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "instance": "10.142.0.11:9100" 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG }, 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG "value": [ 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:48:09 DEBUG 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG [...] 09:48:09 DEBUG { 09:48:09 DEBUG 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG "status": "success", 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG 09:48:09 INFO [http_cmd]: http status code OK 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG "data": { 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG 09:48:09 DEBUG --- status code --- 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG "instance": "10.142.0.51:9100" 09:48:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG "result": [ 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG }, 09:48:09 DEBUG --- http response --- 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG { 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:48:09 DEBUG "value": [ 09:48:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG "metric": { 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG { 09:48:09 DEBUG "pod": "ds-cts-0" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG "data": { 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG "data": { 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG ----- output ----- 09:48:09 DEBUG "result": [ 09:48:09 DEBUG 09:48:09 DEBUG "result": [ 09:48:09 DEBUG { 09:48:09 DEBUG "metric": { 09:48:09 DEBUG { 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG "instance": "10.142.0.11:9100" 09:48:09 DEBUG "metric": { 09:48:09 DEBUG "instance": "10.142.0.11:9100" 09:48:09 DEBUG }, 09:48:09 DEBUG 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:48:09 DEBUG }, 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 09:48:09 DEBUG [...] 09:48:09 DEBUG { 09:48:09 DEBUG [...] 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "0.0010635444140380508" 09:48:09 DEBUG 09:48:09 DEBUG "status": "success", 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG ] 09:48:09 DEBUG 09:48:09 DEBUG "data": { 09:48:09 DEBUG "0.00033333333333350387" 09:48:09 DEBUG 09:48:09 DEBUG } 09:48:09 DEBUG 09:48:09 DEBUG "resultType": "vector", 09:48:09 DEBUG ] 09:48:09 DEBUG "instance": "10.142.0.51:9100" 09:48:09 DEBUG ] 09:48:09 DEBUG "instance": "10.142.0.51:9100" 09:48:09 DEBUG "result": [ 09:48:09 DEBUG } 09:48:09 DEBUG }, 09:48:09 DEBUG } 09:48:09 DEBUG }, 09:48:09 DEBUG { 09:48:09 DEBUG ] 09:48:09 DEBUG "value": [ 09:48:09 DEBUG } 09:48:09 DEBUG "value": [ 09:48:09 DEBUG "metric": { 09:48:09 DEBUG } 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "pod": "ds-cts-0" 09:48:09 DEBUG } 09:48:09 DEBUG "87339.26666666666" 09:48:09 DEBUG "0.00040000000000001514" 09:48:09 DEBUG }, 09:48:09 DEBUG ] 09:48:09 DEBUG ] 09:48:09 DEBUG "value": [ 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 DEBUG 09:48:09 DEBUG ] 09:48:09 DEBUG ] 09:48:09 DEBUG 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 DEBUG 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:09 DEBUG [...] 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG 09:48:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:48:09 DEBUG }, 09:48:09 DEBUG "value": [ 09:48:09 DEBUG 1734774489, 09:48:09 DEBUG "0" 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG ] 09:48:09 DEBUG } 09:48:09 DEBUG } 09:48:51 INFO 09:48:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:48:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:48:51 INFO [loop_until]: OK (rc = 0) 09:48:51 DEBUG --- stdout --- 09:48:51 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:48:51 DEBUG --- stderr --- 09:48:51 DEBUG 09:48:57 INFO 09:48:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:48:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:48:57 INFO [loop_until]: OK (rc = 0) 09:48:57 DEBUG --- stdout --- 09:48:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 107m 0% 3182Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 102m 0% 19859Mi 69% 09:48:57 DEBUG --- stderr --- 09:48:57 DEBUG 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 ERROR [http_cmd]: ERROR 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 ERROR --- status code --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 ERROR --- http response --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774549" 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:49:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "pod": "ds-cts-0" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "603.605042636259" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "pod": "ds-cts-0" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "731299840" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "instance": "10.142.0.11:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "instance": "10.142.0.51:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "68068.83333333334" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "0.042321722814729866" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "instance": "10.142.0.11:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "instance": "10.142.0.51:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "33996.8" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "15775776768" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774549" 09:49:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774549" 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "instance": "10.142.0.11:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "instance": "10.142.0.51:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "85963.03333333333" 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 INFO [http_cmd]: http status code OK 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- status code --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:49:09 DEBUG --- http response --- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG ----- output ----- 09:49:09 DEBUG { 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "status": "success", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "data": { 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG "resultType": "vector", 09:49:09 DEBUG "result": [ 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG "result": [ 09:49:09 DEBUG { 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "pod": "ds-cts-0" 09:49:09 DEBUG "metric": { 09:49:09 DEBUG "instance": "10.142.0.11:9100" 09:49:09 DEBUG "instance": "10.142.0.11:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "pod": "ds-cts-0" 09:49:09 DEBUG }, 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG [...] 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG 09:49:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:49:09 DEBUG 09:49:09 DEBUG "instance": "10.142.0.51:9100" 09:49:09 DEBUG "instance": "10.142.0.51:9100" 09:49:09 DEBUG }, 09:49:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:49:09 DEBUG }, 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG }, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "value": [ 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "0" 09:49:09 DEBUG 1734774549, 09:49:09 DEBUG "0.00043333333333161285" 09:49:09 DEBUG "0.000666666666666534" 09:49:09 DEBUG ] 09:49:09 DEBUG "0.0012004966966721532" 09:49:09 DEBUG ] 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG ] 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:09 DEBUG } 09:49:51 INFO 09:49:51 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:49:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:49:52 INFO [loop_until]: OK (rc = 0) 09:49:52 DEBUG --- stdout --- 09:49:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:49:52 DEBUG --- stderr --- 09:49:52 DEBUG 09:49:57 INFO 09:49:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:49:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:49:57 INFO [loop_until]: OK (rc = 0) 09:49:57 DEBUG --- stdout --- 09:49:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3165Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 103m 0% 19860Mi 69% 09:49:57 DEBUG --- stderr --- 09:49:57 DEBUG 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774609" 09:50:09 ERROR [http_cmd]: ERROR 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 ERROR --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 ERROR --- http response --- 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:50:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:50:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:50:09 DEBUG ----- output ----- 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG { 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG "data": { 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:50:09 DEBUG "result": [ 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "pod": "ds-cts-0" 09:50:09 DEBUG "data": { 09:50:09 DEBUG }, 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "value": [ 09:50:09 DEBUG "result": [ 09:50:09 DEBUG 09:50:09 DEBUG { 09:50:09 DEBUG 09:50:09 DEBUG "metric": { 09:50:09 DEBUG 09:50:09 DEBUG "pod": "ds-cts-0" 09:50:09 DEBUG [...] 09:50:09 DEBUG }, 09:50:09 DEBUG 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:50:09 DEBUG 09:50:09 DEBUG }, 09:50:09 DEBUG [...] 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG 09:50:09 DEBUG "729.2697763177089" 09:50:09 DEBUG 09:50:09 DEBUG ] 09:50:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:50:09 DEBUG } 09:50:09 DEBUG }, 09:50:09 DEBUG ] 09:50:09 DEBUG "value": [ 09:50:09 DEBUG } 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG } 09:50:09 DEBUG "731299840" 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "data": { 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "result": [ 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "instance": "10.142.0.11:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG [...] 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "instance": "10.142.0.51:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG "67397.59999999999" 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "data": { 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "result": [ 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG [...] 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG "0.04489137650851485" 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "data": { 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "result": [ 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "instance": "10.142.0.11:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG [...] 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "instance": "10.142.0.51:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG "23347.2" 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "data": { 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "result": [ 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG [...] 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG "15778148352" 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG } 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774609" 09:50:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774609" 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "data": { 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "result": [ 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "instance": "10.142.0.11:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG [...] 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "instance": "10.142.0.51:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 1734774609, 09:50:09 DEBUG "84930.13333333336" 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG ] 09:50:09 DEBUG } 09:50:09 DEBUG } 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG ----- output ----- 09:50:09 INFO [http_cmd]: http status code OK 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG { 09:50:09 DEBUG --- status code --- 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG "data": { 09:50:09 DEBUG --- http response --- 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:50:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG "result": [ 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG ----- output ----- 09:50:09 DEBUG { 09:50:09 DEBUG { 09:50:09 DEBUG { 09:50:09 DEBUG { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "status": "success", 09:50:09 DEBUG "data": { 09:50:09 DEBUG "pod": "ds-cts-0" 09:50:09 DEBUG "data": { 09:50:09 DEBUG "data": { 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG }, 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "resultType": "vector", 09:50:09 DEBUG "result": [ 09:50:09 DEBUG "value": [ 09:50:09 DEBUG "result": [ 09:50:09 DEBUG "result": [ 09:50:09 DEBUG { 09:50:09 DEBUG 09:50:09 DEBUG { 09:50:09 DEBUG { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "metric": { 09:50:09 DEBUG "instance": "10.142.0.11:9100" 09:50:09 DEBUG 09:50:09 DEBUG "pod": "ds-cts-0" 09:50:09 DEBUG "instance": "10.142.0.11:9100" 09:50:09 DEBUG }, 09:50:09 DEBUG [...] 09:50:09 DEBUG }, 09:50:09 DEBUG }, 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG "value": [ 09:50:09 DEBUG "value": [ 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:50:09 DEBUG 09:50:09 DEBUG 09:50:09 DEBUG [...] 09:50:09 DEBUG }, 09:50:09 DEBUG [...] 09:50:09 DEBUG [...] 09:50:09 DEBUG 09:50:10 DEBUG "value": [ 09:50:10 DEBUG 09:50:10 DEBUG 09:50:10 DEBUG 09:50:10 DEBUG 1734774609, 09:50:10 DEBUG 09:50:10 DEBUG 09:50:10 DEBUG 09:50:10 DEBUG "0.0011041826293748534" 09:50:10 DEBUG 09:50:10 DEBUG 09:50:10 DEBUG "instance": "10.142.0.51:9100" 09:50:10 DEBUG ] 09:50:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:50:10 DEBUG "instance": "10.142.0.51:9100" 09:50:10 DEBUG }, 09:50:10 DEBUG } 09:50:10 DEBUG }, 09:50:10 DEBUG }, 09:50:10 DEBUG "value": [ 09:50:10 DEBUG 1734774609, 09:50:10 DEBUG ] 09:50:10 DEBUG "value": [ 09:50:10 DEBUG "value": [ 09:50:10 DEBUG "0.00040000000000001514" 09:50:10 DEBUG } 09:50:10 DEBUG 1734774609, 09:50:10 DEBUG 1734774609, 09:50:10 DEBUG ] 09:50:10 DEBUG } 09:50:10 DEBUG "0" 09:50:10 DEBUG "0" 09:50:10 DEBUG } 09:50:10 DEBUG ] 09:50:10 DEBUG ] 09:50:10 DEBUG ] 09:50:10 DEBUG } 09:50:10 DEBUG } 09:50:10 DEBUG } 09:50:10 DEBUG ] 09:50:10 DEBUG ] 09:50:10 DEBUG } 09:50:10 DEBUG } 09:50:10 DEBUG } 09:50:10 DEBUG } 09:50:10 DEBUG } 09:50:52 INFO 09:50:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:50:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:50:52 INFO [loop_until]: OK (rc = 0) 09:50:52 DEBUG --- stdout --- 09:50:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 5m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:50:52 DEBUG --- stderr --- 09:50:52 DEBUG 09:50:57 INFO 09:50:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:50:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:50:57 INFO [loop_until]: OK (rc = 0) 09:50:57 DEBUG --- stdout --- 09:50:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3167Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19857Mi 69% 09:50:57 DEBUG --- stderr --- 09:50:57 DEBUG 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 DEBUG --- status code --- 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774669" 09:51:09 DEBUG --- status code --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG --- status code --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- status code --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- http response --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 ERROR [http_cmd]: ERROR 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- status code --- 09:51:09 ERROR --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG --- http response --- 09:51:09 ERROR --- http response --- 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:51:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG ----- output ----- 09:51:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:51:09 DEBUG { 09:51:09 DEBUG ----- output ----- 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG { 09:51:09 DEBUG "data": { 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG "data": { 09:51:09 DEBUG "result": [ 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG { 09:51:09 DEBUG "result": [ 09:51:09 DEBUG "metric": { 09:51:09 DEBUG { 09:51:09 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:51:09 DEBUG "metric": { 09:51:09 DEBUG }, 09:51:09 DEBUG "pod": "ds-cts-0" 09:51:09 DEBUG "value": [ 09:51:09 DEBUG }, 09:51:09 DEBUG 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:51:09 DEBUG 09:51:09 DEBUG }, 09:51:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:51:09 DEBUG "value": [ 09:51:09 DEBUG }, 09:51:09 DEBUG 1734774669, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG "631.1473884392552" 09:51:09 DEBUG 1734774669, 09:51:09 DEBUG ] 09:51:09 DEBUG "731308032" 09:51:09 DEBUG } 09:51:09 DEBUG ] 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:09 DEBUG ----- output ----- 09:51:09 DEBUG { 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG "data": { 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG "result": [ 09:51:09 DEBUG { 09:51:09 DEBUG "metric": { 09:51:09 DEBUG "instance": "10.142.0.11:9100" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG "instance": "10.142.0.51:9100" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 1734774669, 09:51:09 DEBUG "69654.23333333332" 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:09 DEBUG ----- output ----- 09:51:09 DEBUG { 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG "data": { 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG "result": [ 09:51:09 DEBUG { 09:51:09 DEBUG "metric": { 09:51:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 1734774669, 09:51:09 DEBUG "0.03834688390150533" 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774669" 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:09 DEBUG ----- output ----- 09:51:09 DEBUG { 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG "data": { 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG "result": [ 09:51:09 DEBUG { 09:51:09 DEBUG "metric": { 09:51:09 DEBUG "instance": "10.142.0.11:9100" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG "instance": "10.142.0.51:9100" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 1734774669, 09:51:09 DEBUG "17749.333333333332" 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:09 DEBUG ----- output ----- 09:51:09 DEBUG { 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG "data": { 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG "result": [ 09:51:09 DEBUG { 09:51:09 DEBUG "metric": { 09:51:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 1734774669, 09:51:09 DEBUG "15777648640" 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG ] 09:51:09 DEBUG } 09:51:09 DEBUG } 09:51:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774669" 09:51:09 INFO [http_cmd]: http status code OK 09:51:09 DEBUG --- status code --- 09:51:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:09 DEBUG --- http response --- 09:51:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:09 DEBUG ----- output ----- 09:51:09 DEBUG { 09:51:09 DEBUG "status": "success", 09:51:09 DEBUG "data": { 09:51:09 DEBUG "resultType": "vector", 09:51:09 DEBUG "result": [ 09:51:09 DEBUG { 09:51:09 DEBUG "metric": { 09:51:09 DEBUG "instance": "10.142.0.11:9100" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG [...] 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG 09:51:09 DEBUG "instance": "10.142.0.51:9100" 09:51:09 DEBUG }, 09:51:09 DEBUG "value": [ 09:51:09 DEBUG 1734774669, 09:51:10 DEBUG "87724.66666666666" 09:51:10 DEBUG ] 09:51:10 DEBUG } 09:51:10 DEBUG ] 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774670" 09:51:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774670" 09:51:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774670" 09:51:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774670" 09:51:10 INFO [http_cmd]: http status code OK 09:51:10 INFO [http_cmd]: http status code OK 09:51:10 DEBUG --- status code --- 09:51:10 DEBUG --- status code --- 09:51:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:10 INFO [http_cmd]: http status code OK 09:51:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:10 INFO [http_cmd]: http status code OK 09:51:10 DEBUG --- http response --- 09:51:10 DEBUG --- status code --- 09:51:10 DEBUG --- http response --- 09:51:10 DEBUG --- status code --- 09:51:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:51:10 DEBUG ----- output ----- 09:51:10 DEBUG --- http response --- 09:51:10 DEBUG ----- output ----- 09:51:10 DEBUG --- http response --- 09:51:10 DEBUG { 09:51:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:10 DEBUG { 09:51:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:51:10 DEBUG "status": "success", 09:51:10 DEBUG ----- output ----- 09:51:10 DEBUG "status": "success", 09:51:10 DEBUG ----- output ----- 09:51:10 DEBUG "data": { 09:51:10 DEBUG { 09:51:10 DEBUG "data": { 09:51:10 DEBUG { 09:51:10 DEBUG "resultType": "vector", 09:51:10 DEBUG "status": "success", 09:51:10 DEBUG "resultType": "vector", 09:51:10 DEBUG "status": "success", 09:51:10 DEBUG "result": [ 09:51:10 DEBUG "data": { 09:51:10 DEBUG "result": [ 09:51:10 DEBUG "data": { 09:51:10 DEBUG { 09:51:10 DEBUG "resultType": "vector", 09:51:10 DEBUG { 09:51:10 DEBUG "resultType": "vector", 09:51:10 DEBUG "metric": { 09:51:10 DEBUG "result": [ 09:51:10 DEBUG "metric": { 09:51:10 DEBUG "result": [ 09:51:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:51:10 DEBUG { 09:51:10 DEBUG "instance": "10.142.0.11:9100" 09:51:10 DEBUG { 09:51:10 DEBUG }, 09:51:10 DEBUG "metric": { 09:51:10 DEBUG }, 09:51:10 DEBUG "metric": { 09:51:10 DEBUG "value": [ 09:51:10 DEBUG "instance": "10.142.0.11:9100" 09:51:10 DEBUG "value": [ 09:51:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:51:10 DEBUG 09:51:10 DEBUG }, 09:51:10 DEBUG 09:51:10 DEBUG }, 09:51:10 DEBUG 09:51:10 DEBUG "value": [ 09:51:10 DEBUG 09:51:10 DEBUG "value": [ 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG [...] 09:51:10 DEBUG 09:51:10 DEBUG [...] 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG [...] 09:51:10 DEBUG 09:51:10 DEBUG [...] 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG 09:51:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:51:10 DEBUG 09:51:10 DEBUG "instance": "10.142.0.51:9100" 09:51:10 DEBUG 09:51:10 DEBUG }, 09:51:10 DEBUG 09:51:10 DEBUG }, 09:51:10 DEBUG 09:51:10 DEBUG "value": [ 09:51:10 DEBUG "instance": "10.142.0.51:9100" 09:51:10 DEBUG "value": [ 09:51:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:51:10 DEBUG 1734774670, 09:51:10 DEBUG }, 09:51:10 DEBUG 1734774670, 09:51:10 DEBUG }, 09:51:10 DEBUG "0.00030288675313422413" 09:51:10 DEBUG "value": [ 09:51:10 DEBUG "0.00033333333333397756" 09:51:10 DEBUG "value": [ 09:51:10 DEBUG ] 09:51:10 DEBUG 1734774670, 09:51:10 DEBUG ] 09:51:10 DEBUG 1734774670, 09:51:10 DEBUG } 09:51:10 DEBUG "0.00033333333333350387" 09:51:10 DEBUG } 09:51:10 DEBUG "0" 09:51:10 DEBUG ] 09:51:10 DEBUG ] 09:51:10 DEBUG ] 09:51:10 DEBUG ] 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 DEBUG ] 09:51:10 DEBUG } 09:51:10 DEBUG ] 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:10 DEBUG } 09:51:52 INFO 09:51:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:51:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:51:52 INFO [loop_until]: OK (rc = 0) 09:51:52 DEBUG --- stdout --- 09:51:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:51:52 DEBUG --- stderr --- 09:51:52 DEBUG 09:51:57 INFO 09:51:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:51:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:51:57 INFO [loop_until]: OK (rc = 0) 09:51:57 DEBUG --- stdout --- 09:51:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3173Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 97m 0% 19856Mi 69% 09:51:57 DEBUG --- stderr --- 09:51:57 DEBUG 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG --- status code --- 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 ERROR [http_cmd]: ERROR 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 ERROR --- status code --- 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:52:09 ERROR --- http response --- 09:52:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:52:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:52:09 DEBUG ----- output ----- 09:52:09 DEBUG ----- output ----- 09:52:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:52:09 DEBUG { 09:52:09 DEBUG { 09:52:09 DEBUG "status": "success", 09:52:09 DEBUG "status": "success", 09:52:09 DEBUG "data": { 09:52:09 DEBUG "data": { 09:52:09 DEBUG "resultType": "vector", 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774729" 09:52:09 DEBUG "resultType": "vector", 09:52:09 DEBUG "result": [ 09:52:09 DEBUG "result": [ 09:52:09 DEBUG { 09:52:09 DEBUG { 09:52:09 DEBUG "metric": { 09:52:09 DEBUG "metric": { 09:52:09 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:52:09 DEBUG "pod": "ds-cts-0" 09:52:09 DEBUG }, 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG [...] 09:52:09 DEBUG [...] 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG "pod": "ds-cts-0" 09:52:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:52:09 DEBUG }, 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 1734774729, 09:52:09 DEBUG 1734774729, 09:52:09 DEBUG "392.78864595320294" 09:52:09 DEBUG "731308032" 09:52:09 DEBUG ] 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 DEBUG ] 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:09 DEBUG ----- output ----- 09:52:09 DEBUG { 09:52:09 DEBUG "status": "success", 09:52:09 DEBUG "data": { 09:52:09 DEBUG "resultType": "vector", 09:52:09 DEBUG "result": [ 09:52:09 DEBUG { 09:52:09 DEBUG "metric": { 09:52:09 DEBUG "instance": "10.142.0.11:9100" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG [...] 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG "instance": "10.142.0.51:9100" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 1734774729, 09:52:09 DEBUG "68004.16666666666" 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:09 DEBUG ----- output ----- 09:52:09 DEBUG { 09:52:09 DEBUG "status": "success", 09:52:09 DEBUG "data": { 09:52:09 DEBUG "resultType": "vector", 09:52:09 DEBUG "result": [ 09:52:09 DEBUG { 09:52:09 DEBUG "metric": { 09:52:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG [...] 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 1734774729, 09:52:09 DEBUG "0.04161064285095199" 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774729" 09:52:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774729" 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:09 DEBUG ----- output ----- 09:52:09 DEBUG { 09:52:09 DEBUG "status": "success", 09:52:09 DEBUG "data": { 09:52:09 DEBUG "resultType": "vector", 09:52:09 DEBUG "result": [ 09:52:09 DEBUG { 09:52:09 DEBUG "metric": { 09:52:09 DEBUG "instance": "10.142.0.11:9100" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG [...] 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG "instance": "10.142.0.51:9100" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 1734774729, 09:52:09 DEBUG "18568.533333333333" 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:09 INFO [http_cmd]: http status code OK 09:52:09 DEBUG --- status code --- 09:52:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:09 DEBUG --- http response --- 09:52:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:09 DEBUG ----- output ----- 09:52:09 DEBUG { 09:52:09 DEBUG "status": "success", 09:52:09 DEBUG "data": { 09:52:09 DEBUG "resultType": "vector", 09:52:09 DEBUG "result": [ 09:52:09 DEBUG { 09:52:09 DEBUG "metric": { 09:52:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG [...] 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG 09:52:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:52:09 DEBUG }, 09:52:09 DEBUG "value": [ 09:52:09 DEBUG 1734774729, 09:52:09 DEBUG "15779057664" 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG ] 09:52:09 DEBUG } 09:52:09 DEBUG } 09:52:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774730" 09:52:10 INFO [http_cmd]: http status code OK 09:52:10 DEBUG --- status code --- 09:52:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:10 DEBUG --- http response --- 09:52:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:10 DEBUG ----- output ----- 09:52:10 DEBUG { 09:52:10 DEBUG "status": "success", 09:52:10 DEBUG "data": { 09:52:10 DEBUG "resultType": "vector", 09:52:10 DEBUG "result": [ 09:52:10 DEBUG { 09:52:10 DEBUG "metric": { 09:52:10 DEBUG "instance": "10.142.0.11:9100" 09:52:10 DEBUG }, 09:52:10 DEBUG "value": [ 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG [...] 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG "instance": "10.142.0.51:9100" 09:52:10 DEBUG }, 09:52:10 DEBUG "value": [ 09:52:10 DEBUG 1734774730, 09:52:10 DEBUG "85371.33333333333" 09:52:10 DEBUG ] 09:52:10 DEBUG } 09:52:10 DEBUG ] 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774730" 09:52:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774730" 09:52:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774730" 09:52:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774730" 09:52:10 INFO [http_cmd]: http status code OK 09:52:10 DEBUG --- status code --- 09:52:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:10 DEBUG --- http response --- 09:52:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:10 DEBUG ----- output ----- 09:52:10 DEBUG { 09:52:10 DEBUG "status": "success", 09:52:10 DEBUG "data": { 09:52:10 DEBUG "resultType": "vector", 09:52:10 DEBUG "result": [ 09:52:10 DEBUG { 09:52:10 DEBUG "metric": { 09:52:10 INFO [http_cmd]: http status code OK 09:52:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:52:10 INFO [http_cmd]: http status code OK 09:52:10 INFO [http_cmd]: http status code OK 09:52:10 DEBUG --- status code --- 09:52:10 DEBUG }, 09:52:10 DEBUG --- status code --- 09:52:10 DEBUG --- status code --- 09:52:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:10 DEBUG "value": [ 09:52:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:52:10 DEBUG --- http response --- 09:52:10 DEBUG 09:52:10 DEBUG --- http response --- 09:52:10 DEBUG --- http response --- 09:52:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:10 DEBUG 09:52:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:52:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:52:10 DEBUG ----- output ----- 09:52:10 DEBUG 09:52:10 DEBUG ----- output ----- 09:52:10 DEBUG ----- output ----- 09:52:10 DEBUG { 09:52:10 DEBUG [...] 09:52:10 DEBUG { 09:52:10 DEBUG { 09:52:10 DEBUG "status": "success", 09:52:10 DEBUG 09:52:10 DEBUG "status": "success", 09:52:10 DEBUG "status": "success", 09:52:10 DEBUG "data": { 09:52:10 DEBUG 09:52:10 DEBUG "data": { 09:52:10 DEBUG "data": { 09:52:10 DEBUG "resultType": "vector", 09:52:10 DEBUG 09:52:10 DEBUG "resultType": "vector", 09:52:10 DEBUG "resultType": "vector", 09:52:10 DEBUG "result": [ 09:52:10 DEBUG "pod": "ds-cts-0" 09:52:10 DEBUG "result": [ 09:52:10 DEBUG "result": [ 09:52:10 DEBUG { 09:52:10 DEBUG }, 09:52:10 DEBUG { 09:52:10 DEBUG { 09:52:10 DEBUG "metric": { 09:52:10 DEBUG "value": [ 09:52:10 DEBUG "metric": { 09:52:10 DEBUG "metric": { 09:52:10 DEBUG "instance": "10.142.0.11:9100" 09:52:10 DEBUG 1734774730, 09:52:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:52:10 DEBUG "instance": "10.142.0.11:9100" 09:52:10 DEBUG }, 09:52:10 DEBUG "0.003557441503642035" 09:52:10 DEBUG }, 09:52:10 DEBUG }, 09:52:10 DEBUG "value": [ 09:52:10 DEBUG ] 09:52:10 DEBUG "value": [ 09:52:10 DEBUG "value": [ 09:52:10 DEBUG 09:52:10 DEBUG } 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG ] 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG } 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG [...] 09:52:10 DEBUG } 09:52:10 DEBUG [...] 09:52:10 DEBUG [...] 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG 09:52:10 DEBUG "instance": "10.142.0.51:9100" 09:52:10 DEBUG "pod": "ds-cts-0" 09:52:10 DEBUG "instance": "10.142.0.51:9100" 09:52:10 DEBUG }, 09:52:10 DEBUG }, 09:52:10 DEBUG }, 09:52:10 DEBUG "value": [ 09:52:10 DEBUG "value": [ 09:52:10 DEBUG "value": [ 09:52:10 DEBUG 1734774730, 09:52:10 DEBUG 1734774730, 09:52:10 DEBUG 1734774730, 09:52:10 DEBUG "0.001066666666665128" 09:52:10 DEBUG "0" 09:52:10 DEBUG "0.00033333333333350387" 09:52:10 DEBUG ] 09:52:10 DEBUG ] 09:52:10 DEBUG ] 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 DEBUG ] 09:52:10 DEBUG ] 09:52:10 DEBUG ] 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:10 DEBUG } 09:52:52 INFO 09:52:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:52:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:52:52 INFO [loop_until]: OK (rc = 0) 09:52:52 DEBUG --- stdout --- 09:52:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:52:52 DEBUG --- stderr --- 09:52:52 DEBUG 09:52:57 INFO 09:52:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:52:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:52:57 INFO [loop_until]: OK (rc = 0) 09:52:57 DEBUG --- stdout --- 09:52:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 114m 0% 3172Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 101m 0% 19857Mi 69% 09:52:57 DEBUG --- stderr --- 09:52:57 DEBUG 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- status code --- 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 ERROR [http_cmd]: ERROR 09:53:09 ERROR --- status code --- 09:53:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:53:09 ERROR --- http response --- 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- status code --- 09:53:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774789" 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:53:09 DEBUG ----- output ----- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG { 09:53:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG "status": "success", 09:53:09 DEBUG ----- output ----- 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG "data": { 09:53:09 DEBUG { 09:53:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:09 DEBUG "resultType": "vector", 09:53:09 DEBUG "status": "success", 09:53:09 DEBUG ----- output ----- 09:53:09 DEBUG "result": [ 09:53:09 DEBUG "data": { 09:53:09 DEBUG { 09:53:09 DEBUG { 09:53:09 DEBUG "resultType": "vector", 09:53:09 DEBUG "status": "success", 09:53:09 DEBUG "metric": { 09:53:09 DEBUG "result": [ 09:53:09 DEBUG "data": { 09:53:09 DEBUG "pod": "ds-idrepo-0" 09:53:09 DEBUG { 09:53:09 DEBUG "resultType": "vector", 09:53:09 DEBUG }, 09:53:09 DEBUG "metric": { 09:53:09 DEBUG "result": [ 09:53:09 DEBUG "value": [ 09:53:09 DEBUG "pod": "ds-cts-0" 09:53:09 DEBUG { 09:53:09 DEBUG 09:53:09 DEBUG }, 09:53:09 DEBUG "metric": { 09:53:09 DEBUG 09:53:09 DEBUG "value": [ 09:53:09 DEBUG "instance": "10.142.0.11:9100" 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG }, 09:53:09 DEBUG [...] 09:53:09 DEBUG 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG [...] 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG "pod": "ds-cts-0" 09:53:09 DEBUG 09:53:09 DEBUG [...] 09:53:09 DEBUG }, 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG "value": [ 09:53:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:53:09 DEBUG 09:53:09 DEBUG 1734774789, 09:53:09 DEBUG }, 09:53:09 DEBUG 09:53:09 DEBUG "402.8192099655795" 09:53:09 DEBUG "value": [ 09:53:09 DEBUG "instance": "10.142.0.51:9100" 09:53:09 DEBUG ] 09:53:09 DEBUG 1734774789, 09:53:09 DEBUG }, 09:53:09 DEBUG } 09:53:09 DEBUG "731308032" 09:53:09 DEBUG "value": [ 09:53:09 DEBUG ] 09:53:09 DEBUG ] 09:53:09 DEBUG 1734774789, 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:09 DEBUG "69731.6" 09:53:09 DEBUG } 09:53:09 DEBUG ] 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:09 DEBUG ----- output ----- 09:53:09 DEBUG { 09:53:09 DEBUG "status": "success", 09:53:09 DEBUG "data": { 09:53:09 DEBUG "resultType": "vector", 09:53:09 DEBUG "result": [ 09:53:09 DEBUG { 09:53:09 DEBUG "metric": { 09:53:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:53:09 DEBUG }, 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG [...] 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:53:09 DEBUG }, 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 1734774789, 09:53:09 DEBUG "0.04146693732030446" 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774789" 09:53:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774789" 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:09 DEBUG ----- output ----- 09:53:09 DEBUG { 09:53:09 DEBUG "status": "success", 09:53:09 DEBUG "data": { 09:53:09 DEBUG "resultType": "vector", 09:53:09 DEBUG "result": [ 09:53:09 DEBUG { 09:53:09 DEBUG "metric": { 09:53:09 DEBUG "instance": "10.142.0.11:9100" 09:53:09 DEBUG }, 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG [...] 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG "instance": "10.142.0.51:9100" 09:53:09 DEBUG }, 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 1734774789, 09:53:09 DEBUG "30583.466666666667" 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:09 INFO [http_cmd]: http status code OK 09:53:09 DEBUG --- status code --- 09:53:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:09 DEBUG --- http response --- 09:53:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:09 DEBUG ----- output ----- 09:53:09 DEBUG { 09:53:09 DEBUG "status": "success", 09:53:09 DEBUG "data": { 09:53:09 DEBUG "resultType": "vector", 09:53:09 DEBUG "result": [ 09:53:09 DEBUG { 09:53:09 DEBUG "metric": { 09:53:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:53:09 DEBUG }, 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG [...] 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG 09:53:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:53:09 DEBUG }, 09:53:09 DEBUG "value": [ 09:53:09 DEBUG 1734774789, 09:53:09 DEBUG "15775592448" 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG ] 09:53:09 DEBUG } 09:53:09 DEBUG } 09:53:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774790" 09:53:10 INFO [http_cmd]: http status code OK 09:53:10 DEBUG --- status code --- 09:53:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:10 DEBUG --- http response --- 09:53:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:10 DEBUG ----- output ----- 09:53:10 DEBUG { 09:53:10 DEBUG "status": "success", 09:53:10 DEBUG "data": { 09:53:10 DEBUG "resultType": "vector", 09:53:10 DEBUG "result": [ 09:53:10 DEBUG { 09:53:10 DEBUG "metric": { 09:53:10 DEBUG "instance": "10.142.0.11:9100" 09:53:10 DEBUG }, 09:53:10 DEBUG "value": [ 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG [...] 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG "instance": "10.142.0.51:9100" 09:53:10 DEBUG }, 09:53:10 DEBUG "value": [ 09:53:10 DEBUG 1734774790, 09:53:10 DEBUG "86995.93333333335" 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG } 09:53:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774790" 09:53:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774790" 09:53:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774790" 09:53:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774790" 09:53:10 INFO [http_cmd]: http status code OK 09:53:10 DEBUG --- status code --- 09:53:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:10 DEBUG --- http response --- 09:53:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:53:10 DEBUG ----- output ----- 09:53:10 DEBUG { 09:53:10 DEBUG "status": "success", 09:53:10 DEBUG "data": { 09:53:10 DEBUG "resultType": "vector", 09:53:10 DEBUG "result": [ 09:53:10 DEBUG { 09:53:10 DEBUG "metric": { 09:53:10 DEBUG "pod": "ds-idrepo-0" 09:53:10 DEBUG }, 09:53:10 DEBUG "value": [ 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG [...] 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG "pod": "ds-cts-0" 09:53:10 DEBUG }, 09:53:10 DEBUG "value": [ 09:53:10 DEBUG 1734774790, 09:53:10 DEBUG "0.003499885264714375" 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG } 09:53:10 INFO [http_cmd]: http status code OK 09:53:10 DEBUG --- status code --- 09:53:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:10 DEBUG --- http response --- 09:53:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:10 INFO [http_cmd]: http status code OK 09:53:10 DEBUG ----- output ----- 09:53:10 INFO [http_cmd]: http status code OK 09:53:10 DEBUG --- status code --- 09:53:10 DEBUG { 09:53:10 DEBUG --- status code --- 09:53:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:10 DEBUG "status": "success", 09:53:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:53:10 DEBUG --- http response --- 09:53:10 DEBUG "data": { 09:53:10 DEBUG --- http response --- 09:53:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:53:10 DEBUG "resultType": "vector", 09:53:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:53:10 DEBUG ----- output ----- 09:53:10 DEBUG "result": [ 09:53:10 DEBUG ----- output ----- 09:53:10 DEBUG { 09:53:10 DEBUG { 09:53:10 DEBUG { 09:53:10 DEBUG "status": "success", 09:53:10 DEBUG "data": { 09:53:10 DEBUG "status": "success", 09:53:10 DEBUG "resultType": "vector", 09:53:10 DEBUG "result": [ 09:53:10 DEBUG "data": { 09:53:10 DEBUG "metric": { 09:53:10 DEBUG { 09:53:10 DEBUG "metric": { 09:53:10 DEBUG "pod": "ds-idrepo-0" 09:53:10 DEBUG "instance": "10.142.0.11:9100" 09:53:10 DEBUG "resultType": "vector", 09:53:10 DEBUG }, 09:53:10 DEBUG "value": [ 09:53:10 DEBUG "result": [ 09:53:10 DEBUG }, 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG "value": [ 09:53:10 DEBUG { 09:53:10 DEBUG 09:53:10 DEBUG [...] 09:53:10 DEBUG "metric": { 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG "instance": "10.142.0.11:9100" 09:53:10 DEBUG 09:53:10 DEBUG "pod": "ds-cts-0" 09:53:10 DEBUG }, 09:53:10 DEBUG 09:53:10 DEBUG }, 09:53:10 DEBUG "value": [ 09:53:10 DEBUG [...] 09:53:10 DEBUG "value": [ 09:53:10 DEBUG 1734774790, 09:53:10 DEBUG "0" 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG 09:53:10 DEBUG 09:53:10 DEBUG } 09:53:10 DEBUG [...] 09:53:10 DEBUG "instance": "10.142.0.51:9100" 09:53:10 DEBUG 09:53:10 DEBUG }, 09:53:10 DEBUG 09:53:10 DEBUG "value": [ 09:53:10 DEBUG 09:53:10 DEBUG 1734774790, 09:53:10 DEBUG "instance": "10.142.0.51:9100" 09:53:10 DEBUG "0.000333333333334925" 09:53:10 DEBUG }, 09:53:10 DEBUG ] 09:53:10 DEBUG "value": [ 09:53:10 DEBUG } 09:53:10 DEBUG 1734774790, 09:53:10 DEBUG ] 09:53:10 DEBUG "0" 09:53:10 DEBUG } 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG } 09:53:10 DEBUG ] 09:53:10 DEBUG } 09:53:10 DEBUG } 09:53:52 INFO 09:53:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:53:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:53:52 INFO [loop_until]: OK (rc = 0) 09:53:52 DEBUG --- stdout --- 09:53:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:53:52 DEBUG --- stderr --- 09:53:52 DEBUG 09:53:57 INFO 09:53:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:53:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:53:57 INFO [loop_until]: OK (rc = 0) 09:53:57 DEBUG --- stdout --- 09:53:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 117m 0% 3171Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 108m 0% 19866Mi 69% 09:53:57 DEBUG --- stderr --- 09:53:57 DEBUG 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG --- http response --- 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 ERROR [http_cmd]: ERROR 09:54:09 ERROR --- status code --- 09:54:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:54:09 ERROR --- http response --- 09:54:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:54:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774849" 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- http response --- 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:54:09 DEBUG --- status code --- 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG ----- output ----- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG { 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG "status": "success", 09:54:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG "data": { 09:54:09 DEBUG ----- output ----- 09:54:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:54:09 DEBUG "resultType": "vector", 09:54:09 DEBUG { 09:54:09 DEBUG ----- output ----- 09:54:09 DEBUG "result": [ 09:54:09 DEBUG "status": "success", 09:54:09 DEBUG { 09:54:09 DEBUG { 09:54:09 DEBUG "data": { 09:54:09 DEBUG "status": "success", 09:54:09 DEBUG "metric": { 09:54:09 DEBUG "resultType": "vector", 09:54:09 DEBUG "data": { 09:54:09 DEBUG "pod": "ds-cts-0" 09:54:09 DEBUG "result": [ 09:54:09 DEBUG "resultType": "vector", 09:54:09 DEBUG }, 09:54:09 DEBUG { 09:54:09 DEBUG "result": [ 09:54:09 DEBUG "value": [ 09:54:09 DEBUG "metric": { 09:54:09 DEBUG { 09:54:09 DEBUG 09:54:09 DEBUG "instance": "10.142.0.11:9100" 09:54:09 DEBUG "metric": { 09:54:09 DEBUG 09:54:09 DEBUG }, 09:54:09 DEBUG "pod": "ds-cts-0" 09:54:09 DEBUG 09:54:09 DEBUG "value": [ 09:54:09 DEBUG }, 09:54:09 DEBUG [...] 09:54:09 DEBUG 09:54:09 DEBUG "value": [ 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 INFO [http_cmd]: http status code OK 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 DEBUG [...] 09:54:09 DEBUG --- status code --- 09:54:09 DEBUG 09:54:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:54:09 DEBUG 09:54:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:09 DEBUG [...] 09:54:09 DEBUG }, 09:54:09 DEBUG 09:54:09 DEBUG --- http response --- 09:54:09 DEBUG 09:54:09 DEBUG "value": [ 09:54:09 DEBUG 09:54:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:09 DEBUG 09:54:09 DEBUG 1734774849, 09:54:09 DEBUG "instance": "10.142.0.51:9100" 09:54:09 DEBUG ----- output ----- 09:54:09 DEBUG 09:54:09 DEBUG "731308032" 09:54:09 DEBUG }, 09:54:09 DEBUG { 09:54:09 DEBUG "pod": "ds-idrepo-0" 09:54:09 DEBUG ] 09:54:09 DEBUG "value": [ 09:54:09 DEBUG "status": "success", 09:54:09 DEBUG }, 09:54:09 DEBUG } 09:54:09 DEBUG 1734774849, 09:54:09 DEBUG "data": { 09:54:09 DEBUG "value": [ 09:54:09 DEBUG ] 09:54:09 DEBUG "67538.13333333333" 09:54:09 DEBUG "resultType": "vector", 09:54:09 DEBUG 1734774849, 09:54:09 DEBUG } 09:54:09 DEBUG ] 09:54:09 DEBUG "result": [ 09:54:09 DEBUG "855.9061464521328" 09:54:09 DEBUG } 09:54:09 DEBUG } 09:54:09 DEBUG { 09:54:09 DEBUG "metric": { 09:54:09 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:54:09 DEBUG }, 09:54:09 DEBUG ] 09:54:09 DEBUG ] 09:54:09 DEBUG "value": [ 09:54:09 DEBUG } 09:54:09 DEBUG } 09:54:09 DEBUG 09:54:09 DEBUG } 09:54:09 DEBUG ] 09:54:09 DEBUG 09:54:09 DEBUG } 09:54:09 DEBUG 09:54:09 DEBUG } 09:54:09 DEBUG [...] 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 DEBUG 09:54:09 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:54:09 DEBUG }, 09:54:09 DEBUG "value": [ 09:54:09 DEBUG 1734774849, 09:54:09 DEBUG "0.04611627119824534" 09:54:09 DEBUG ] 09:54:09 DEBUG } 09:54:09 DEBUG ] 09:54:09 DEBUG } 09:54:09 DEBUG } 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774849" 09:54:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774849" 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "instance": "10.142.0.11:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "instance": "10.142.0.51:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774849, 09:54:10 DEBUG "18841.6" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774849, 09:54:10 DEBUG "15773310976" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774850" 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "instance": "10.142.0.11:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "instance": "10.142.0.51:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774850, 09:54:10 DEBUG "86016.46666666666" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774850" 09:54:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774850" 09:54:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774850" 09:54:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774850" 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "pod": "ds-cts-0" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "pod": "ds-idrepo-0" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774850, 09:54:10 DEBUG "0.005113137705287883" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "pod": "ds-cts-0" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "pod": "ds-idrepo-0" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774850, 09:54:10 DEBUG "0" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "instance": "10.142.0.11:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "instance": "10.142.0.51:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774850, 09:54:10 DEBUG "0.000666666666666534" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:10 INFO [http_cmd]: http status code OK 09:54:10 DEBUG --- status code --- 09:54:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:54:10 DEBUG --- http response --- 09:54:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:54:10 DEBUG ----- output ----- 09:54:10 DEBUG { 09:54:10 DEBUG "status": "success", 09:54:10 DEBUG "data": { 09:54:10 DEBUG "resultType": "vector", 09:54:10 DEBUG "result": [ 09:54:10 DEBUG { 09:54:10 DEBUG "metric": { 09:54:10 DEBUG "instance": "10.142.0.11:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG [...] 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG 09:54:10 DEBUG "instance": "10.142.0.51:9100" 09:54:10 DEBUG }, 09:54:10 DEBUG "value": [ 09:54:10 DEBUG 1734774850, 09:54:10 DEBUG "0.00036666666666652263" 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG ] 09:54:10 DEBUG } 09:54:10 DEBUG } 09:54:52 INFO 09:54:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:54:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:54:52 INFO [loop_until]: OK (rc = 0) 09:54:52 DEBUG --- stdout --- 09:54:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 526Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:54:52 DEBUG --- stderr --- 09:54:52 DEBUG 09:54:57 INFO 09:54:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:54:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:54:57 INFO [loop_until]: OK (rc = 0) 09:54:57 DEBUG --- stdout --- 09:54:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3166Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 104m 0% 19861Mi 69% 09:54:57 DEBUG --- stderr --- 09:54:57 DEBUG 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774909" 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 ERROR [http_cmd]: ERROR 09:55:09 ERROR --- status code --- 09:55:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 ERROR --- http response --- 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 DEBUG --- status code --- 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774909" 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774909" 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774909" 09:55:09 DEBUG --- http response --- 09:55:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774909" 09:55:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:55:09 INFO [http_cmd]: http status code OK 09:55:09 DEBUG --- status code --- 09:55:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:09 DEBUG --- http response --- 09:55:09 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:55:09 DEBUG ----- output ----- 09:55:09 DEBUG { 09:55:09 DEBUG "status": "success", 09:55:09 DEBUG "data": { 09:55:09 DEBUG "resultType": "vector", 09:55:09 DEBUG "result": [ 09:55:09 DEBUG { 09:55:09 DEBUG "metric": { 09:55:09 DEBUG "pod": "ds-cts-0" 09:55:09 DEBUG }, 09:55:09 DEBUG "value": [ 09:55:09 DEBUG 09:55:09 DEBUG 09:55:09 DEBUG 09:55:09 DEBUG [...] 09:55:09 DEBUG 09:55:09 DEBUG 09:55:09 DEBUG 09:55:09 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:55:09 DEBUG }, 09:55:09 DEBUG "value": [ 09:55:09 DEBUG 1734774909, 09:55:09 DEBUG "731308032" 09:55:09 DEBUG ] 09:55:09 DEBUG } 09:55:09 DEBUG ] 09:55:09 DEBUG } 09:55:09 DEBUG } 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG { 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG "status": "success", 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG "data": { 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG "result": [ 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG { 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG "metric": { 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG "pod": "ds-idrepo-0" 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG { 09:55:10 DEBUG }, 09:55:10 DEBUG { 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "value": [ 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "data": { 09:55:10 DEBUG 09:55:10 DEBUG "data": { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG "result": [ 09:55:10 DEBUG 09:55:10 DEBUG "result": [ 09:55:10 DEBUG { 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG "metric": { 09:55:10 DEBUG { 09:55:10 DEBUG 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774910" 09:55:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:55:10 DEBUG }, 09:55:10 DEBUG "metric": { 09:55:10 DEBUG 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:55:10 DEBUG "instance": "10.142.0.11:9100" 09:55:10 DEBUG 09:55:10 DEBUG }, 09:55:10 DEBUG }, 09:55:10 DEBUG 09:55:10 DEBUG "value": [ 09:55:10 DEBUG "value": [ 09:55:10 DEBUG [...] 09:55:10 DEBUG 1734774909, 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG "519.1698543895167" 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:55:10 DEBUG [...] 09:55:10 DEBUG } 09:55:10 DEBUG }, 09:55:10 DEBUG 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG 1734774909, 09:55:10 DEBUG 09:55:10 DEBUG "0.03804103914259344" 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774910" 09:55:10 DEBUG "instance": "10.142.0.51:9100" 09:55:10 DEBUG ] 09:55:10 DEBUG }, 09:55:10 DEBUG } 09:55:10 DEBUG "value": [ 09:55:10 DEBUG ] 09:55:10 DEBUG 1734774909, 09:55:10 DEBUG } 09:55:10 DEBUG "67343.03333333335" 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG { 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "data": { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG "result": [ 09:55:10 DEBUG { 09:55:10 DEBUG "metric": { 09:55:10 DEBUG "instance": "10.142.0.11:9100" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG "instance": "10.142.0.51:9100" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG "24439.466666666667" 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG { 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "data": { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG "result": [ 09:55:10 DEBUG { 09:55:10 DEBUG "metric": { 09:55:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG "15774810112" 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774910" 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG { 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "data": { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG "result": [ 09:55:10 DEBUG { 09:55:10 DEBUG "metric": { 09:55:10 DEBUG "instance": "10.142.0.11:9100" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG "instance": "10.142.0.51:9100" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG "85323.26666666669" 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774910" 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774910" 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774910" 09:55:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774910" 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG --- http response --- 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG { 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG { 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "data": { 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG "data": { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG "result": [ 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG "result": [ 09:55:10 DEBUG { 09:55:10 DEBUG "data": { 09:55:10 DEBUG { 09:55:10 DEBUG "metric": { 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG "metric": { 09:55:10 DEBUG "pod": "ds-idrepo-0" 09:55:10 DEBUG "result": [ 09:55:10 DEBUG "pod": "ds-idrepo-0" 09:55:10 DEBUG }, 09:55:10 DEBUG { 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG "metric": { 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG "instance": "10.142.0.11:9100" 09:55:10 INFO [http_cmd]: http status code OK 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG }, 09:55:10 DEBUG --- status code --- 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG "value": [ 09:55:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:55:10 DEBUG 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG --- http response --- 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG ----- output ----- 09:55:10 DEBUG 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG { 09:55:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 09:55:10 DEBUG 09:55:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:55:10 DEBUG "status": "success", 09:55:10 DEBUG }, 09:55:10 DEBUG 09:55:10 DEBUG }, 09:55:10 DEBUG "data": { 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 09:55:10 DEBUG "value": [ 09:55:10 DEBUG "resultType": "vector", 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG "instance": "10.142.0.51:9100" 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG "result": [ 09:55:10 DEBUG "0" 09:55:10 DEBUG }, 09:55:10 DEBUG "0" 09:55:10 DEBUG { 09:55:10 DEBUG ] 09:55:10 DEBUG "value": [ 09:55:10 DEBUG ] 09:55:10 DEBUG "metric": { 09:55:10 DEBUG } 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG } 09:55:10 DEBUG "instance": "10.142.0.11:9100" 09:55:10 DEBUG ] 09:55:10 DEBUG "0.000666666666666534" 09:55:10 DEBUG ] 09:55:10 DEBUG }, 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG "value": [ 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:10 DEBUG 09:55:10 DEBUG ] 09:55:10 DEBUG 09:55:10 DEBUG } 09:55:10 DEBUG 09:55:10 DEBUG } 09:55:10 DEBUG [...] 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG 09:55:10 DEBUG "instance": "10.142.0.51:9100" 09:55:10 DEBUG }, 09:55:10 DEBUG "value": [ 09:55:10 DEBUG 1734774910, 09:55:10 DEBUG "0.0005333333333339852" 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG ] 09:55:10 DEBUG } 09:55:10 DEBUG } 09:55:52 INFO 09:55:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:55:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:55:52 INFO [loop_until]: OK (rc = 0) 09:55:52 DEBUG --- stdout --- 09:55:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 526Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:55:52 DEBUG --- stderr --- 09:55:52 DEBUG 09:55:57 INFO 09:55:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:55:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:55:57 INFO [loop_until]: OK (rc = 0) 09:55:57 DEBUG --- stdout --- 09:55:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 111m 0% 3162Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 98m 0% 19860Mi 69% 09:55:57 DEBUG --- stderr --- 09:55:57 DEBUG 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734774969" 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 DEBUG --- status code --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- http response --- 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774969" 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:09 ERROR [http_cmd]: ERROR 09:56:09 ERROR --- status code --- 09:56:09 INFO [http_cmd]: http status code OK 09:56:09 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:56:09 DEBUG --- status code --- 09:56:09 ERROR --- http response --- 09:56:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:09 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:56:09 DEBUG --- http response --- 09:56:09 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:56:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734774969" 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774970" 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG { 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "data": { 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "result": [ 09:56:10 DEBUG { 09:56:10 DEBUG "metric": { 09:56:10 DEBUG "pod": "ds-cts-0" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 1734774969, 09:56:10 DEBUG "731316224" 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734774970" 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774970" 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774970" 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG { 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "data": { 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "result": [ 09:56:10 DEBUG { 09:56:10 DEBUG "metric": { 09:56:10 DEBUG "pod": "ds-cts-0" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "pod": "ds-idrepo-0" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG "628.4559027644646" 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734774970" 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG { 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG { 09:56:10 DEBUG "data": { 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "data": { 09:56:10 DEBUG "result": [ 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG { 09:56:10 DEBUG "result": [ 09:56:10 DEBUG "metric": { 09:56:10 DEBUG { 09:56:10 DEBUG "instance": "10.142.0.11:9100" 09:56:10 DEBUG "metric": { 09:56:10 DEBUG }, 09:56:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:56:10 DEBUG "value": [ 09:56:10 DEBUG }, 09:56:10 DEBUG 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "instance": "10.142.0.51:9100" 09:56:10 DEBUG 09:56:10 DEBUG }, 09:56:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:56:10 DEBUG "value": [ 09:56:10 DEBUG }, 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG "68192.9" 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG ] 09:56:10 DEBUG "0.03862903762285999" 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG { 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "data": { 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "result": [ 09:56:10 DEBUG { 09:56:10 DEBUG "metric": { 09:56:10 DEBUG "instance": "10.142.0.11:9100" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "instance": "10.142.0.51:9100" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG "26350.933333333334" 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG { 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "data": { 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "result": [ 09:56:10 DEBUG { 09:56:10 DEBUG "metric": { 09:56:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG "15778779136" 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774970" 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG { 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "data": { 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "result": [ 09:56:10 DEBUG { 09:56:10 DEBUG "metric": { 09:56:10 DEBUG "instance": "10.142.0.11:9100" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG [...] 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "instance": "10.142.0.51:9100" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG "85452.73333333334" 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774970" 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734774970" 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774970" 09:56:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734774970" 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG --- http response --- 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG { 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG "data": { 09:56:10 DEBUG { 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "result": [ 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:56:10 DEBUG "data": { 09:56:10 DEBUG { 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "metric": { 09:56:10 DEBUG { 09:56:10 DEBUG "result": [ 09:56:10 DEBUG "pod": "ds-cts-0" 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG { 09:56:10 DEBUG }, 09:56:10 DEBUG "data": { 09:56:10 DEBUG "metric": { 09:56:10 DEBUG "value": [ 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "instance": "10.142.0.11:9100" 09:56:10 DEBUG 09:56:10 DEBUG "result": [ 09:56:10 DEBUG }, 09:56:10 DEBUG 09:56:10 DEBUG { 09:56:10 DEBUG "value": [ 09:56:10 INFO [http_cmd]: http status code OK 09:56:10 DEBUG 09:56:10 DEBUG "metric": { 09:56:10 DEBUG 09:56:10 DEBUG --- status code --- 09:56:10 DEBUG [...] 09:56:10 DEBUG "pod": "ds-cts-0" 09:56:10 DEBUG 09:56:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:56:10 DEBUG 09:56:10 DEBUG }, 09:56:10 DEBUG 09:56:10 DEBUG --- http response --- 09:56:10 DEBUG 09:56:10 DEBUG "value": [ 09:56:10 DEBUG [...] 09:56:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG ----- output ----- 09:56:10 DEBUG "pod": "ds-idrepo-0" 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG { 09:56:10 DEBUG }, 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "status": "success", 09:56:10 DEBUG "value": [ 09:56:10 DEBUG [...] 09:56:10 DEBUG "instance": "10.142.0.51:9100" 09:56:10 DEBUG "data": { 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG 09:56:10 DEBUG }, 09:56:10 DEBUG "resultType": "vector", 09:56:10 DEBUG "0" 09:56:10 DEBUG 09:56:10 DEBUG "value": [ 09:56:10 DEBUG "result": [ 09:56:10 DEBUG ] 09:56:10 DEBUG 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG { 09:56:10 DEBUG } 09:56:10 DEBUG "pod": "ds-idrepo-0" 09:56:10 DEBUG "0.0003333333333330302" 09:56:10 DEBUG "metric": { 09:56:10 DEBUG ] 09:56:10 DEBUG }, 09:56:10 DEBUG ] 09:56:10 DEBUG "instance": "10.142.0.11:9100" 09:56:10 DEBUG } 09:56:10 DEBUG "value": [ 09:56:10 DEBUG } 09:56:10 DEBUG }, 09:56:10 DEBUG } 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG ] 09:56:10 DEBUG "value": [ 09:56:10 DEBUG "0.003380635530663454" 09:56:10 DEBUG } 09:56:10 DEBUG 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG 09:56:10 DEBUG } 09:56:10 DEBUG 09:56:10 DEBUG ] 09:56:10 DEBUG [...] 09:56:10 DEBUG } 09:56:10 DEBUG 09:56:10 DEBUG } 09:56:10 DEBUG 09:56:10 DEBUG 09:56:10 DEBUG "instance": "10.142.0.51:9100" 09:56:10 DEBUG }, 09:56:10 DEBUG "value": [ 09:56:10 DEBUG 1734774970, 09:56:10 DEBUG "0.00043333333333350765" 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG ] 09:56:10 DEBUG } 09:56:10 DEBUG } 09:56:52 INFO 09:56:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:56:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:56:52 INFO [loop_until]: OK (rc = 0) 09:56:52 DEBUG --- stdout --- 09:56:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 09:56:52 DEBUG --- stderr --- 09:56:52 DEBUG 09:56:57 INFO 09:56:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:56:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:56:57 INFO [loop_until]: OK (rc = 0) 09:56:57 DEBUG --- stdout --- 09:56:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3174Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 102m 0% 19863Mi 69% 09:56:57 DEBUG --- stderr --- 09:56:57 DEBUG 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:09 INFO [http_cmd]: http status code OK 09:57:09 DEBUG --- status code --- 09:57:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:09 DEBUG --- http response --- 09:57:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775029" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775030" 09:57:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775030" 09:57:10 ERROR [http_cmd]: ERROR 09:57:10 ERROR --- status code --- 09:57:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:57:10 ERROR --- http response --- 09:57:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:57:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775030" 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775030" 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "data": { 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "result": [ 09:57:10 DEBUG { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "pod": "ds-cts-0" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG [...] 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG "731316224" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775030" 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775030" 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "data": { 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "result": [ 09:57:10 DEBUG { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "pod": "ds-cts-0" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG [...] 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG "248.18225884634026" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "data": { 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "result": [ 09:57:10 DEBUG { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "instance": "10.142.0.11:9100" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG [...] 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "instance": "10.142.0.51:9100" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG "66892.10000000002" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "data": { 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "result": [ 09:57:10 DEBUG { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG [...] 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG "0.038497770948084355" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG { 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG "data": { 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG "result": [ 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "instance": "10.142.0.11:9100" 09:57:10 DEBUG "data": { 09:57:10 DEBUG }, 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "value": [ 09:57:10 DEBUG "result": [ 09:57:10 DEBUG 09:57:10 DEBUG { 09:57:10 DEBUG 09:57:10 DEBUG "metric": { 09:57:10 DEBUG 09:57:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:57:10 DEBUG [...] 09:57:10 DEBUG }, 09:57:10 DEBUG 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "instance": "10.142.0.51:9100" 09:57:10 DEBUG 09:57:10 DEBUG }, 09:57:10 DEBUG [...] 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG 09:57:10 DEBUG "33860.26666666666" 09:57:10 DEBUG 09:57:10 DEBUG ] 09:57:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:57:10 DEBUG } 09:57:10 DEBUG }, 09:57:10 DEBUG ] 09:57:10 DEBUG "value": [ 09:57:10 DEBUG } 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG } 09:57:10 DEBUG "15778574336" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775030" 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "data": { 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "result": [ 09:57:10 DEBUG { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "instance": "10.142.0.11:9100" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG [...] 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "instance": "10.142.0.51:9100" 09:57:10 DEBUG }, 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG "84856.33333333334" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775030" 09:57:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775030" 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG { 09:57:10 DEBUG { 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG "data": { 09:57:10 DEBUG "data": { 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG "result": [ 09:57:10 DEBUG "result": [ 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:57:10 DEBUG { 09:57:10 DEBUG { 09:57:10 DEBUG ----- output ----- 09:57:10 INFO [http_cmd]: http status code OK 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "metric": { 09:57:10 DEBUG { 09:57:10 DEBUG --- status code --- 09:57:10 DEBUG "instance": "10.142.0.11:9100" 09:57:10 DEBUG "pod": "ds-cts-0" 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:57:10 DEBUG }, 09:57:10 DEBUG }, 09:57:10 DEBUG "data": { 09:57:10 DEBUG --- http response --- 09:57:10 DEBUG "value": [ 09:57:10 DEBUG "value": [ 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "result": [ 09:57:10 DEBUG ----- output ----- 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG { 09:57:10 DEBUG { 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "metric": { 09:57:10 DEBUG "status": "success", 09:57:10 DEBUG [...] 09:57:10 DEBUG [...] 09:57:10 DEBUG "pod": "ds-cts-0" 09:57:10 DEBUG "data": { 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG }, 09:57:10 DEBUG "resultType": "vector", 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG "value": [ 09:57:10 DEBUG "result": [ 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG { 09:57:10 DEBUG "instance": "10.142.0.51:9100" 09:57:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:57:10 DEBUG 09:57:10 DEBUG "metric": { 09:57:10 DEBUG }, 09:57:10 DEBUG }, 09:57:10 DEBUG 09:57:10 DEBUG "instance": "10.142.0.11:9100" 09:57:10 DEBUG "value": [ 09:57:10 DEBUG "value": [ 09:57:10 DEBUG [...] 09:57:10 DEBUG }, 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG 09:57:10 DEBUG "value": [ 09:57:10 DEBUG "0.001000000000000038" 09:57:10 DEBUG "0" 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG ] 09:57:10 DEBUG ] 09:57:10 DEBUG 09:57:10 DEBUG 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:57:10 DEBUG 09:57:10 DEBUG ] 09:57:10 DEBUG ] 09:57:10 DEBUG }, 09:57:10 DEBUG [...] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 DEBUG "value": [ 09:57:10 DEBUG 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG 09:57:10 DEBUG "0.0009334916189571421" 09:57:10 DEBUG 09:57:10 DEBUG ] 09:57:10 DEBUG "instance": "10.142.0.51:9100" 09:57:10 DEBUG } 09:57:10 DEBUG }, 09:57:10 DEBUG ] 09:57:10 DEBUG "value": [ 09:57:10 DEBUG } 09:57:10 DEBUG 1734775030, 09:57:10 DEBUG } 09:57:10 DEBUG "0.001133333333332113" 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG ] 09:57:10 DEBUG } 09:57:10 DEBUG } 09:57:52 INFO 09:57:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:57:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:57:52 INFO [loop_until]: OK (rc = 0) 09:57:52 DEBUG --- stdout --- 09:57:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 4m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:57:52 DEBUG --- stderr --- 09:57:52 DEBUG 09:57:57 INFO 09:57:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:57:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:57:57 INFO [loop_until]: OK (rc = 0) 09:57:57 DEBUG --- stdout --- 09:57:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3174Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 100m 0% 19862Mi 69% 09:57:57 DEBUG --- stderr --- 09:57:57 DEBUG 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- http response --- 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: http status code OK 09:58:09 DEBUG --- status code --- 09:58:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:09 DEBUG --- http response --- 09:58:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775089" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775090" 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775090" 09:58:10 ERROR [http_cmd]: ERROR 09:58:10 ERROR --- status code --- 09:58:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:58:10 ERROR --- http response --- 09:58:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:58:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775090" 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG { 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "data": { 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "result": [ 09:58:10 DEBUG { 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "pod": "ds-cts-0" 09:58:10 DEBUG }, 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775090" 09:58:10 DEBUG }, 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG "731332608" 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775090" 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG --- http response --- 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775090" 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG { 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 DEBUG { 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "data": { 09:58:10 DEBUG { 09:58:10 DEBUG "data": { 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "result": [ 09:58:10 DEBUG "data": { 09:58:10 DEBUG "result": [ 09:58:10 DEBUG { 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG { 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "result": [ 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "instance": "10.142.0.11:9100" 09:58:10 DEBUG { 09:58:10 DEBUG "pod": "ds-cts-0" 09:58:10 DEBUG }, 09:58:10 DEBUG "metric": { 09:58:10 DEBUG }, 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG }, 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG "instance": "10.142.0.51:9100" 09:58:10 DEBUG 09:58:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:58:10 DEBUG }, 09:58:10 DEBUG 09:58:10 DEBUG }, 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG }, 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG "69255.40000000001" 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "0" 09:58:10 DEBUG ] 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG "0.05009812157508848" 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG ] 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG { 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG "data": { 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG "result": [ 09:58:10 DEBUG { 09:58:10 DEBUG { 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "data": { 09:58:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG }, 09:58:10 DEBUG "result": [ 09:58:10 DEBUG "value": [ 09:58:10 DEBUG { 09:58:10 DEBUG 09:58:10 DEBUG "metric": { 09:58:10 DEBUG 09:58:10 DEBUG "instance": "10.142.0.11:9100" 09:58:10 DEBUG 09:58:10 DEBUG }, 09:58:10 DEBUG [...] 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:58:10 DEBUG [...] 09:58:10 DEBUG }, 09:58:10 DEBUG 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG 09:58:10 DEBUG "15780020224" 09:58:10 DEBUG "instance": "10.142.0.51:9100" 09:58:10 DEBUG ] 09:58:10 DEBUG }, 09:58:10 DEBUG } 09:58:10 DEBUG "value": [ 09:58:10 DEBUG ] 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG } 09:58:10 DEBUG "24712.533333333333" 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775090" 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG { 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "data": { 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "result": [ 09:58:10 DEBUG { 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "instance": "10.142.0.11:9100" 09:58:10 DEBUG }, 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG "instance": "10.142.0.51:9100" 09:58:10 DEBUG }, 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG "86940.90000000002" 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775090" 09:58:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775090" 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG { 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "data": { 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "result": [ 09:58:10 DEBUG { 09:58:10 DEBUG "metric": { 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG "instance": "10.142.0.11:9100" 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG }, 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG "value": [ 09:58:10 INFO [http_cmd]: http status code OK 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG 09:58:10 DEBUG --- status code --- 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:58:10 DEBUG 09:58:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG 09:58:10 DEBUG --- http response --- 09:58:10 DEBUG { 09:58:10 DEBUG { 09:58:10 DEBUG [...] 09:58:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG 09:58:10 DEBUG ----- output ----- 09:58:10 DEBUG "data": { 09:58:10 DEBUG "data": { 09:58:10 DEBUG 09:58:10 DEBUG { 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG 09:58:10 DEBUG "status": "success", 09:58:10 DEBUG "result": [ 09:58:10 DEBUG "result": [ 09:58:10 DEBUG "instance": "10.142.0.51:9100" 09:58:10 DEBUG "data": { 09:58:10 DEBUG { 09:58:10 DEBUG { 09:58:10 DEBUG }, 09:58:10 DEBUG "resultType": "vector", 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "result": [ 09:58:10 DEBUG "instance": "10.142.0.11:9100" 09:58:10 DEBUG "pod": "ds-cts-0" 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG { 09:58:10 DEBUG }, 09:58:10 DEBUG }, 09:58:10 DEBUG "0" 09:58:10 DEBUG "metric": { 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "value": [ 09:58:10 DEBUG ] 09:58:10 DEBUG "pod": "ds-cts-0" 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG } 09:58:10 DEBUG }, 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG ] 09:58:10 DEBUG "value": [ 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG } 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG [...] 09:58:10 DEBUG } 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG [...] 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG 09:58:10 DEBUG "instance": "10.142.0.51:9100" 09:58:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:58:10 DEBUG 09:58:10 DEBUG }, 09:58:10 DEBUG }, 09:58:10 DEBUG 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "value": [ 09:58:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG }, 09:58:10 DEBUG "0.0004666666666688949" 09:58:10 DEBUG "0.0015455353028026348" 09:58:10 DEBUG "value": [ 09:58:10 DEBUG ] 09:58:10 DEBUG ] 09:58:10 DEBUG 1734775090, 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG "0" 09:58:10 DEBUG ] 09:58:10 DEBUG ] 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:10 DEBUG ] 09:58:10 DEBUG } 09:58:10 DEBUG } 09:58:52 INFO 09:58:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:58:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:58:52 INFO [loop_until]: OK (rc = 0) 09:58:52 DEBUG --- stdout --- 09:58:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:58:52 DEBUG --- stderr --- 09:58:52 DEBUG 09:58:57 INFO 09:58:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:58:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:58:57 INFO [loop_until]: OK (rc = 0) 09:58:57 DEBUG --- stdout --- 09:58:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3176Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 19863Mi 69% 09:58:57 DEBUG --- stderr --- 09:58:57 DEBUG 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775149" 09:59:09 INFO [http_cmd]: http status code OK 09:59:09 DEBUG --- status code --- 09:59:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:09 DEBUG --- http response --- 09:59:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- http response --- 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775150" 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775150" 09:59:10 ERROR [http_cmd]: ERROR 09:59:10 ERROR --- status code --- 09:59:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 09:59:10 ERROR --- http response --- 09:59:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 09:59:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG { 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG "data": { 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "result": [ 09:59:10 DEBUG { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "pod": "ds-cts-0" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "731332608" 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775150" 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG { 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG "data": { 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG { 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG "result": [ 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG "data": { 09:59:10 DEBUG { 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "metric": { 09:59:10 DEBUG { 09:59:10 DEBUG "result": [ 09:59:10 DEBUG "pod": "ds-cts-0" 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG { 09:59:10 DEBUG }, 09:59:10 DEBUG "data": { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "value": [ 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:59:10 DEBUG 09:59:10 DEBUG "result": [ 09:59:10 DEBUG }, 09:59:10 DEBUG 09:59:10 DEBUG { 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG "metric": { 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "instance": "10.142.0.11:9100" 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG }, 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG "value": [ 09:59:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG }, 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:59:10 DEBUG [...] 09:59:10 DEBUG "269.3850707004275" 09:59:10 DEBUG }, 09:59:10 DEBUG 09:59:10 DEBUG ] 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG "value": [ 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG 09:59:10 DEBUG } 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG 09:59:10 DEBUG ] 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG "0.044561543591100976" 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG "instance": "10.142.0.51:9100" 09:59:10 DEBUG } 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG ] 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG }, 09:59:10 DEBUG } 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG } 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG "value": [ 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG ] 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG { 09:59:10 DEBUG } 09:59:10 DEBUG { 09:59:10 DEBUG "67505.13333333335" 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG } 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG ] 09:59:10 DEBUG "data": { 09:59:10 DEBUG "data": { 09:59:10 DEBUG } 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG ] 09:59:10 DEBUG "result": [ 09:59:10 DEBUG "result": [ 09:59:10 DEBUG } 09:59:10 DEBUG { 09:59:10 DEBUG { 09:59:10 DEBUG } 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 09:59:10 DEBUG "instance": "10.142.0.11:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 09:59:10 DEBUG "instance": "10.142.0.51:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "15778660352" 09:59:10 DEBUG "26077.866666666665" 09:59:10 DEBUG ] 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775150" 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG { 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG "data": { 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "result": [ 09:59:10 DEBUG { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "instance": "10.142.0.11:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "instance": "10.142.0.51:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "85867.16666666666" 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775150" 09:59:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775150" 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG { 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG "data": { 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "result": [ 09:59:10 DEBUG { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "instance": "10.142.0.11:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "instance": "10.142.0.51:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "0" 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 INFO [http_cmd]: http status code OK 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:59:10 DEBUG --- status code --- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG { 09:59:10 DEBUG --- http response --- 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG "data": { 09:59:10 DEBUG ----- output ----- 09:59:10 DEBUG { 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG { 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "data": { 09:59:10 DEBUG "status": "success", 09:59:10 DEBUG "result": [ 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "data": { 09:59:10 DEBUG { 09:59:10 DEBUG "result": [ 09:59:10 DEBUG "resultType": "vector", 09:59:10 DEBUG "result": [ 09:59:10 DEBUG { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG { 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "pod": "ds-cts-0" 09:59:10 DEBUG "metric": { 09:59:10 DEBUG "pod": "ds-cts-0" 09:59:10 DEBUG }, 09:59:10 DEBUG "instance": "10.142.0.11:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG [...] 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG 09:59:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:59:10 DEBUG 09:59:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 09:59:10 DEBUG }, 09:59:10 DEBUG "instance": "10.142.0.51:9100" 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG }, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "value": [ 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "0.0005147977638984423" 09:59:10 DEBUG 1734775150, 09:59:10 DEBUG "0" 09:59:10 DEBUG ] 09:59:10 DEBUG "0.0005999999999990754" 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG ] 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:10 DEBUG } 09:59:52 INFO 09:59:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 09:59:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:59:52 INFO [loop_until]: OK (rc = 0) 09:59:52 DEBUG --- stdout --- 09:59:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 09:59:52 DEBUG --- stderr --- 09:59:52 DEBUG 09:59:57 INFO 09:59:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 09:59:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 09:59:57 INFO [loop_until]: OK (rc = 0) 09:59:57 DEBUG --- stdout --- 09:59:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 119m 0% 3182Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 97m 0% 19864Mi 69% 09:59:57 DEBUG --- stderr --- 09:59:57 DEBUG 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG --- status code --- 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775209" 10:00:09 INFO [http_cmd]: http status code OK 10:00:09 DEBUG --- status code --- 10:00:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:09 DEBUG --- http response --- 10:00:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775210" 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 ERROR [http_cmd]: ERROR 10:00:10 DEBUG --- http response --- 10:00:10 ERROR --- status code --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:00:10 ERROR --- http response --- 10:00:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:00:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG { 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG "data": { 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "result": [ 10:00:10 DEBUG { 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "pod": "ds-cts-0" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG "731332608" 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775210" 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG { 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG "data": { 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "result": [ 10:00:10 DEBUG { 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG "804.2352688121466" 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG { 10:00:10 DEBUG { 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG "data": { 10:00:10 DEBUG "data": { 10:00:10 DEBUG { 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "resultType": "vector", 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG "result": [ 10:00:10 DEBUG "result": [ 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG "data": { 10:00:10 DEBUG { 10:00:10 DEBUG { 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "metric": { 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG "result": [ 10:00:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:00:10 DEBUG "instance": "10.142.0.11:9100" 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG { 10:00:10 DEBUG }, 10:00:10 DEBUG }, 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "value": [ 10:00:10 DEBUG "value": [ 10:00:10 DEBUG { 10:00:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG }, 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "data": { 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG [...] 10:00:10 DEBUG "result": [ 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG { 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "metric": { 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "instance": "10.142.0.11:9100" 10:00:10 DEBUG 10:00:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:00:10 DEBUG "instance": "10.142.0.51:9100" 10:00:10 DEBUG }, 10:00:10 DEBUG 10:00:10 DEBUG }, 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG "value": [ 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG 10:00:10 DEBUG }, 10:00:10 DEBUG "0.046018679552951254" 10:00:10 DEBUG "67533.16666666666" 10:00:10 DEBUG 10:00:10 DEBUG "value": [ 10:00:10 DEBUG ] 10:00:10 DEBUG ] 10:00:10 DEBUG [...] 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG 10:00:10 DEBUG "15780769792" 10:00:10 DEBUG ] 10:00:10 DEBUG ] 10:00:10 DEBUG 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG "instance": "10.142.0.51:9100" 10:00:10 DEBUG ] 10:00:10 DEBUG }, 10:00:10 DEBUG } 10:00:10 DEBUG "value": [ 10:00:10 DEBUG } 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG "18432" 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775210" 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG { 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG "data": { 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "result": [ 10:00:10 DEBUG { 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "instance": "10.142.0.11:9100" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "instance": "10.142.0.51:9100" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG "85835.33333333333" 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775210" 10:00:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775210" 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG { 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG "data": { 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "result": [ 10:00:10 DEBUG { 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "instance": "10.142.0.11:9100" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "instance": "10.142.0.51:9100" 10:00:10 DEBUG }, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG "0" 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG --- http response --- 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 INFO [http_cmd]: http status code OK 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG --- status code --- 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:00:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:00:10 DEBUG { 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG --- http response --- 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG { 10:00:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:00:10 DEBUG "data": { 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG ----- output ----- 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "data": { 10:00:10 DEBUG { 10:00:10 DEBUG "result": [ 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "status": "success", 10:00:10 DEBUG { 10:00:10 DEBUG "result": [ 10:00:10 DEBUG "data": { 10:00:10 DEBUG "metric": { 10:00:10 DEBUG { 10:00:10 DEBUG "resultType": "vector", 10:00:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:00:10 DEBUG "metric": { 10:00:10 DEBUG "result": [ 10:00:10 DEBUG }, 10:00:10 DEBUG "instance": "10.142.0.11:9100" 10:00:10 DEBUG { 10:00:10 DEBUG "value": [ 10:00:10 DEBUG }, 10:00:10 DEBUG "metric": { 10:00:10 DEBUG 10:00:10 DEBUG "value": [ 10:00:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG }, 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG "value": [ 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG [...] 10:00:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:00:10 DEBUG 10:00:10 DEBUG 10:00:10 DEBUG }, 10:00:10 DEBUG "instance": "10.142.0.51:9100" 10:00:10 DEBUG 10:00:10 DEBUG "value": [ 10:00:10 DEBUG }, 10:00:10 DEBUG 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG "value": [ 10:00:10 DEBUG "pod": "ds-cts-0" 10:00:10 DEBUG "0" 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG }, 10:00:10 DEBUG ] 10:00:10 DEBUG "0.00039999999999906777" 10:00:10 DEBUG "value": [ 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG 1734775210, 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG "0" 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:10 DEBUG ] 10:00:10 DEBUG } 10:00:10 DEBUG } 10:00:52 INFO 10:00:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:00:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:00:52 INFO [loop_until]: OK (rc = 0) 10:00:52 DEBUG --- stdout --- 10:00:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:00:52 DEBUG --- stderr --- 10:00:52 DEBUG 10:00:57 INFO 10:00:57 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:00:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:00:58 INFO [loop_until]: OK (rc = 0) 10:00:58 DEBUG --- stdout --- 10:00:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 121m 0% 3164Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19860Mi 69% 10:00:58 DEBUG --- stderr --- 10:00:58 DEBUG 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775269" 10:01:09 INFO [http_cmd]: http status code OK 10:01:09 DEBUG --- status code --- 10:01:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:09 DEBUG --- http response --- 10:01:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 ERROR [http_cmd]: ERROR 10:01:10 DEBUG --- status code --- 10:01:10 ERROR --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:01:10 ERROR --- http response --- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG { 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "data": { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "result": [ 10:01:10 DEBUG { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "pod": "ds-cts-0" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG "731332608" 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG { 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "data": { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "result": [ 10:01:10 DEBUG { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "pod": "ds-idrepo-0" 10:01:10 DEBUG }, 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775270" 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775270" 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 1734775270, 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775270" 10:01:10 DEBUG "233.73904242280474" 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG { 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "data": { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "result": [ 10:01:10 DEBUG { 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG "metric": { 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG "instance": "10.142.0.11:9100" 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG }, 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG "value": [ 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG 10:01:10 DEBUG { 10:01:10 DEBUG { 10:01:10 DEBUG { 10:01:10 DEBUG 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG [...] 10:01:10 DEBUG "data": { 10:01:10 DEBUG "data": { 10:01:10 DEBUG "data": { 10:01:10 DEBUG 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG 10:01:10 DEBUG "result": [ 10:01:10 DEBUG "result": [ 10:01:10 DEBUG "result": [ 10:01:10 DEBUG 10:01:10 DEBUG { 10:01:10 DEBUG { 10:01:10 DEBUG { 10:01:10 DEBUG "instance": "10.142.0.51:9100" 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG }, 10:01:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:01:10 DEBUG "instance": "10.142.0.11:9100" 10:01:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:01:10 DEBUG "value": [ 10:01:10 DEBUG }, 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775270" 10:01:10 DEBUG }, 10:01:10 DEBUG }, 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "67643.66666666667" 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG ] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG } 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG ] 10:01:10 DEBUG [...] 10:01:10 DEBUG [...] 10:01:10 DEBUG [...] 10:01:10 DEBUG } 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG } 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:01:10 DEBUG "instance": "10.142.0.51:9100" 10:01:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:01:10 DEBUG }, 10:01:10 DEBUG }, 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG "0.04756853958728216" 10:01:10 DEBUG "306380.8" 10:01:10 DEBUG "15780319232" 10:01:10 DEBUG ] 10:01:10 DEBUG ] 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG ] 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG { 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "data": { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "result": [ 10:01:10 DEBUG { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "instance": "10.142.0.11:9100" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG "instance": "10.142.0.51:9100" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG "86000.16666666664" 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775270" 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG { 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "data": { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "result": [ 10:01:10 DEBUG { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "instance": "10.142.0.11:9100" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG "instance": "10.142.0.51:9100" 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG "0.001333333333333068" 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775270" 10:01:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775270" 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 INFO [http_cmd]: http status code OK 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG --- status code --- 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG --- http response --- 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:01:10 DEBUG { 10:01:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG ----- output ----- 10:01:10 DEBUG { 10:01:10 DEBUG "data": { 10:01:10 DEBUG { 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "status": "success", 10:01:10 DEBUG "data": { 10:01:10 DEBUG "result": [ 10:01:10 DEBUG "data": { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG { 10:01:10 DEBUG "resultType": "vector", 10:01:10 DEBUG "result": [ 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "result": [ 10:01:10 DEBUG { 10:01:10 DEBUG "pod": "ds-idrepo-0" 10:01:10 DEBUG { 10:01:10 DEBUG "metric": { 10:01:10 DEBUG }, 10:01:10 DEBUG "metric": { 10:01:10 DEBUG "instance": "10.142.0.11:9100" 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "pod": "ds-idrepo-0" 10:01:10 DEBUG }, 10:01:10 DEBUG 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG [...] 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:01:10 DEBUG 10:01:10 DEBUG 10:01:10 DEBUG }, 10:01:10 DEBUG 10:01:10 DEBUG "instance": "10.142.0.51:9100" 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:01:10 DEBUG }, 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG }, 10:01:10 DEBUG "value": [ 10:01:10 DEBUG "0.0007914855031608438" 10:01:10 DEBUG "value": [ 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG ] 10:01:10 DEBUG 1734775270, 10:01:10 DEBUG "0.0014333333333307033" 10:01:10 DEBUG } 10:01:10 DEBUG "0" 10:01:10 DEBUG ] 10:01:10 DEBUG ] 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG ] 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:10 DEBUG } 10:01:52 INFO 10:01:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:01:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:01:52 INFO [loop_until]: OK (rc = 0) 10:01:52 DEBUG --- stdout --- 10:01:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:01:52 DEBUG --- stderr --- 10:01:52 DEBUG 10:01:58 INFO 10:01:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:01:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:01:58 INFO [loop_until]: OK (rc = 0) 10:01:58 DEBUG --- stdout --- 10:01:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 118m 0% 3132Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 108m 0% 19863Mi 69% 10:01:58 DEBUG --- stderr --- 10:01:58 DEBUG 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- http response --- 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG --- http response --- 10:02:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775329" 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:09 INFO [http_cmd]: http status code OK 10:02:09 DEBUG --- status code --- 10:02:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:09 DEBUG --- http response --- 10:02:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775330" 10:02:10 ERROR [http_cmd]: ERROR 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 ERROR --- status code --- 10:02:10 DEBUG --- status code --- 10:02:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 ERROR --- http response --- 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG { 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "data": { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG { 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "pod": "ds-cts-0" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "731398144" 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG { 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "data": { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG { 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "pod": "ds-cts-0" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "0" 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775330" 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG { 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG { 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG "data": { 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "data": { 10:02:10 DEBUG { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "data": { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG "data": { 10:02:10 DEBUG { 10:02:10 DEBUG "result": [ 10:02:10 DEBUG { 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "metric": { 10:02:10 DEBUG { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:02:10 DEBUG "instance": "10.142.0.11:9100" 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "result": [ 10:02:10 DEBUG }, 10:02:10 DEBUG }, 10:02:10 DEBUG "instance": "10.142.0.11:9100" 10:02:10 DEBUG { 10:02:10 DEBUG "value": [ 10:02:10 DEBUG "value": [ 10:02:10 DEBUG }, 10:02:10 DEBUG "metric": { 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "value": [ 10:02:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG }, 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG "value": [ 10:02:10 DEBUG [...] 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG [...] 10:02:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:02:10 DEBUG "instance": "10.142.0.51:9100" 10:02:10 DEBUG 10:02:10 DEBUG { 10:02:10 DEBUG 10:02:10 DEBUG }, 10:02:10 DEBUG }, 10:02:10 DEBUG "instance": "10.142.0.51:9100" 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG 10:02:10 DEBUG "value": [ 10:02:10 DEBUG "value": [ 10:02:10 DEBUG }, 10:02:10 DEBUG "data": { 10:02:10 DEBUG 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:02:10 DEBUG "0.04478235748511806" 10:02:10 DEBUG "71568.8" 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "result": [ 10:02:10 DEBUG }, 10:02:10 DEBUG ] 10:02:10 DEBUG ] 10:02:10 DEBUG "18022.4" 10:02:10 DEBUG { 10:02:10 DEBUG "value": [ 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG "metric": { 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG ] 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG "instance": "10.142.0.11:9100" 10:02:10 DEBUG "15777386496" 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG }, 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG "value": [ 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG 10:02:10 DEBUG ] 10:02:10 DEBUG 10:02:10 DEBUG } 10:02:10 DEBUG 10:02:10 DEBUG } 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "instance": "10.142.0.51:9100" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "89356.83333333334" 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775330" 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG { 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "data": { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG { 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "instance": "10.142.0.11:9100" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "instance": "10.142.0.51:9100" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "0" 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775330" 10:02:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775330" 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG { 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "data": { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG { 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "pod": "ds-cts-0" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:02:10 DEBUG }, 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG "0" 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG --- http response --- 10:02:10 INFO [http_cmd]: http status code OK 10:02:10 DEBUG --- status code --- 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:02:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG --- http response --- 10:02:10 DEBUG { 10:02:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG ----- output ----- 10:02:10 DEBUG "data": { 10:02:10 DEBUG { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "status": "success", 10:02:10 DEBUG "result": [ 10:02:10 DEBUG "data": { 10:02:10 DEBUG { 10:02:10 DEBUG "resultType": "vector", 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "result": [ 10:02:10 DEBUG "instance": "10.142.0.11:9100" 10:02:10 DEBUG { 10:02:10 DEBUG }, 10:02:10 DEBUG "metric": { 10:02:10 DEBUG "value": [ 10:02:10 DEBUG "pod": "ds-cts-0" 10:02:10 DEBUG 10:02:10 DEBUG }, 10:02:10 DEBUG 10:02:10 DEBUG "value": [ 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG [...] 10:02:10 DEBUG 10:02:10 DEBUG 10:02:10 DEBUG "instance": "10.142.0.51:9100" 10:02:10 DEBUG 10:02:10 DEBUG }, 10:02:10 DEBUG 10:02:10 DEBUG "value": [ 10:02:10 DEBUG "pod": "ds-idrepo-0" 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG }, 10:02:10 DEBUG "0.0004333333333325603" 10:02:10 DEBUG "value": [ 10:02:10 DEBUG ] 10:02:10 DEBUG 1734775330, 10:02:10 DEBUG } 10:02:10 DEBUG "0" 10:02:10 DEBUG ] 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:10 DEBUG ] 10:02:10 DEBUG } 10:02:10 DEBUG } 10:02:52 INFO 10:02:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:02:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:02:52 INFO [loop_until]: OK (rc = 0) 10:02:52 DEBUG --- stdout --- 10:02:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 3m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:02:52 DEBUG --- stderr --- 10:02:52 DEBUG 10:02:58 INFO 10:02:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:02:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:02:58 INFO [loop_until]: OK (rc = 0) 10:02:58 DEBUG --- stdout --- 10:02:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3159Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 103m 0% 19862Mi 69% 10:02:58 DEBUG --- stderr --- 10:02:58 DEBUG 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775389" 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:09 INFO [http_cmd]: http status code OK 10:03:09 DEBUG --- status code --- 10:03:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:09 DEBUG --- http response --- 10:03:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 ERROR [http_cmd]: ERROR 10:03:10 ERROR --- status code --- 10:03:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:03:10 ERROR --- http response --- 10:03:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:03:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG { 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "data": { 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "result": [ 10:03:10 DEBUG { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "pod": "ds-cts-0" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG [...] 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG "732119040" 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG { 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "data": { 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "result": [ 10:03:10 DEBUG { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "pod": "ds-idrepo-0" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG [...] 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG "1278.3405830579688" 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775390" 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG { 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "data": { 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG "result": [ 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG { 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG "metric": { 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 DEBUG "instance": "10.142.0.11:9100" 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG }, 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG "value": [ 10:03:10 DEBUG { 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG "data": { 10:03:10 DEBUG "data": { 10:03:10 DEBUG 10:03:10 DEBUG "data": { 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "result": [ 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG "result": [ 10:03:10 DEBUG [...] 10:03:10 DEBUG "result": [ 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG 10:03:10 DEBUG { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "metric": { 10:03:10 DEBUG 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "instance": "10.142.0.11:9100" 10:03:10 DEBUG "data": { 10:03:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:03:10 DEBUG 10:03:10 DEBUG "instance": "10.142.0.11:9100" 10:03:10 DEBUG }, 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG }, 10:03:10 DEBUG "instance": "10.142.0.51:9100" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG "result": [ 10:03:10 DEBUG "value": [ 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG { 10:03:10 DEBUG 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG "metric": { 10:03:10 DEBUG 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:03:10 DEBUG 10:03:10 DEBUG "87468.3" 10:03:10 DEBUG 10:03:10 DEBUG [...] 10:03:10 DEBUG }, 10:03:10 DEBUG [...] 10:03:10 DEBUG ] 10:03:10 DEBUG [...] 10:03:10 DEBUG 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG } 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG ] 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG } 10:03:10 DEBUG 10:03:10 DEBUG "instance": "10.142.0.51:9100" 10:03:10 DEBUG 10:03:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:03:10 DEBUG } 10:03:10 DEBUG "instance": "10.142.0.51:9100" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG }, 10:03:10 DEBUG }, 10:03:10 DEBUG [...] 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG "26487.466666666667" 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG 10:03:10 DEBUG ] 10:03:10 DEBUG "0.038491034759123924" 10:03:10 DEBUG "69774.76666666666" 10:03:10 DEBUG 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG ] 10:03:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG }, 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG ] 10:03:10 DEBUG "value": [ 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG "15779102720" 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775390" 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG { 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "data": { 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "result": [ 10:03:10 DEBUG { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "instance": "10.142.0.11:9100" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG [...] 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG "instance": "10.142.0.51:9100" 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG "0.00033333333333350387" 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775390" 10:03:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775390" 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 INFO [http_cmd]: http status code OK 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG --- status code --- 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG --- http response --- 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:03:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG ----- output ----- 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "status": "success", 10:03:10 DEBUG "data": { 10:03:10 DEBUG "data": { 10:03:10 DEBUG "data": { 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "resultType": "vector", 10:03:10 DEBUG "result": [ 10:03:10 DEBUG "result": [ 10:03:10 DEBUG "result": [ 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "metric": { 10:03:10 DEBUG "pod": "ds-idrepo-0" 10:03:10 DEBUG "instance": "10.142.0.11:9100" 10:03:10 DEBUG "pod": "ds-idrepo-0" 10:03:10 DEBUG }, 10:03:10 DEBUG }, 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG "value": [ 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG [...] 10:03:10 DEBUG [...] 10:03:10 DEBUG [...] 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG 10:03:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:03:10 DEBUG "instance": "10.142.0.51:9100" 10:03:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:03:10 DEBUG }, 10:03:10 DEBUG }, 10:03:10 DEBUG }, 10:03:10 DEBUG "value": [ 10:03:10 DEBUG "value": [ 10:03:10 DEBUG "value": [ 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG 1734775390, 10:03:10 DEBUG "0" 10:03:10 DEBUG "0.0004333333333325603" 10:03:10 DEBUG "0.001914682837620599" 10:03:10 DEBUG ] 10:03:10 DEBUG ] 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG ] 10:03:10 DEBUG ] 10:03:10 DEBUG ] 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:10 DEBUG } 10:03:52 INFO 10:03:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:03:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:03:52 INFO [loop_until]: OK (rc = 0) 10:03:52 DEBUG --- stdout --- 10:03:52 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:03:52 DEBUG --- stderr --- 10:03:52 DEBUG 10:03:58 INFO 10:03:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:03:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:03:58 INFO [loop_until]: OK (rc = 0) 10:03:58 DEBUG --- stdout --- 10:03:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3161Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19857Mi 69% 10:03:58 DEBUG --- stderr --- 10:03:58 DEBUG 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775449" 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:09 INFO [http_cmd]: http status code OK 10:04:09 DEBUG --- status code --- 10:04:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:09 DEBUG --- http response --- 10:04:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 ERROR [http_cmd]: ERROR 10:04:10 ERROR --- status code --- 10:04:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:04:10 ERROR --- http response --- 10:04:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:04:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- http response --- 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:04:10 DEBUG { 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "data": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "pod": "ds-cts-0" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "732135424" 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG { 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "data": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "pod": "ds-idrepo-0" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "pod": "ds-cts-0" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "0" 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775450" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG { 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "data": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "instance": "10.142.0.11:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "instance": "10.142.0.51:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "85705.59999999999" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG ] 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG } 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG ] 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG } 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG } 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG { 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "data": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "instance": "10.142.0.11:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "instance": "10.142.0.51:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "19114.666666666664" 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- http response --- 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG "data": { 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG { 10:04:10 DEBUG { 10:04:10 DEBUG "result": [ 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG { 10:04:10 DEBUG "data": { 10:04:10 DEBUG "data": { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:04:10 DEBUG "result": [ 10:04:10 DEBUG "result": [ 10:04:10 DEBUG }, 10:04:10 DEBUG { 10:04:10 DEBUG { 10:04:10 DEBUG "value": [ 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG 10:04:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:04:10 DEBUG "instance": "10.142.0.11:9100" 10:04:10 DEBUG 10:04:10 DEBUG }, 10:04:10 DEBUG }, 10:04:10 DEBUG 10:04:10 DEBUG "value": [ 10:04:10 DEBUG "value": [ 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG [...] 10:04:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG }, 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:04:10 DEBUG "instance": "10.142.0.51:9100" 10:04:10 DEBUG "0.051978731413270196" 10:04:10 DEBUG }, 10:04:10 DEBUG }, 10:04:10 DEBUG ] 10:04:10 DEBUG "value": [ 10:04:10 DEBUG "value": [ 10:04:10 DEBUG } 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG ] 10:04:10 DEBUG "15775358976" 10:04:10 DEBUG "67696.06666666667" 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775450" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG { 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "data": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "instance": "10.142.0.11:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "instance": "10.142.0.51:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "0" 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775450" 10:04:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775450" 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG { 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG "data": { 10:04:10 DEBUG { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG "data": { 10:04:10 DEBUG { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "result": [ 10:04:10 DEBUG "pod": "ds-idrepo-0" 10:04:10 DEBUG { 10:04:10 DEBUG }, 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "value": [ 10:04:10 DEBUG "pod": "ds-idrepo-0" 10:04:10 DEBUG 10:04:10 DEBUG }, 10:04:10 DEBUG 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:04:10 DEBUG 10:04:10 DEBUG }, 10:04:10 DEBUG 10:04:10 DEBUG "value": [ 10:04:10 DEBUG "pod": "ds-cts-0" 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG }, 10:04:10 DEBUG "0" 10:04:10 DEBUG "value": [ 10:04:10 DEBUG ] 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG } 10:04:10 DEBUG "0" 10:04:10 DEBUG ] 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:10 INFO [http_cmd]: http status code OK 10:04:10 DEBUG --- status code --- 10:04:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:04:10 DEBUG --- http response --- 10:04:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:04:10 DEBUG ----- output ----- 10:04:10 DEBUG { 10:04:10 DEBUG "status": "success", 10:04:10 DEBUG "data": { 10:04:10 DEBUG "resultType": "vector", 10:04:10 DEBUG "result": [ 10:04:10 DEBUG { 10:04:10 DEBUG "metric": { 10:04:10 DEBUG "instance": "10.142.0.11:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG [...] 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG 10:04:10 DEBUG "instance": "10.142.0.51:9100" 10:04:10 DEBUG }, 10:04:10 DEBUG "value": [ 10:04:10 DEBUG 1734775450, 10:04:10 DEBUG "0.00040000000000001514" 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG ] 10:04:10 DEBUG } 10:04:10 DEBUG } 10:04:52 INFO 10:04:52 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:04:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:04:53 INFO [loop_until]: OK (rc = 0) 10:04:53 DEBUG --- stdout --- 10:04:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:04:53 DEBUG --- stderr --- 10:04:53 DEBUG 10:04:58 INFO 10:04:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:04:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:04:58 INFO [loop_until]: OK (rc = 0) 10:04:58 DEBUG --- stdout --- 10:04:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 112m 0% 3205Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 95m 0% 19861Mi 69% 10:04:58 DEBUG --- stderr --- 10:04:58 DEBUG 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775509" 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:09 INFO [http_cmd]: http status code OK 10:05:09 DEBUG --- status code --- 10:05:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:09 DEBUG --- http response --- 10:05:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775510" 10:05:10 ERROR [http_cmd]: ERROR 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 ERROR --- status code --- 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- status code --- 10:05:10 ERROR --- http response --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 DEBUG --- http response --- 10:05:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 DEBUG { 10:05:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "data": { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG { 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "pod": "ds-cts-0" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG "732135424" 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG { 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "data": { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG { 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "pod": "ds-cts-0" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG "387.9400157853197" 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775510" 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG "data": { 10:05:10 DEBUG { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG "data": { 10:05:10 DEBUG { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "result": [ 10:05:10 DEBUG "instance": "10.142.0.11:9100" 10:05:10 DEBUG { 10:05:10 DEBUG }, 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "value": [ 10:05:10 DEBUG "instance": "10.142.0.11:9100" 10:05:10 DEBUG 10:05:10 DEBUG }, 10:05:10 DEBUG 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "instance": "10.142.0.51:9100" 10:05:10 DEBUG 10:05:10 DEBUG }, 10:05:10 DEBUG 10:05:10 DEBUG "value": [ 10:05:10 DEBUG "instance": "10.142.0.51:9100" 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG }, 10:05:10 DEBUG "31266.13333333333" 10:05:10 DEBUG "value": [ 10:05:10 DEBUG ] 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG } 10:05:10 DEBUG "85853.49999999999" 10:05:10 DEBUG ] 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG { 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "data": { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG { 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG "0.04345115501370902" 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG { 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG "data": { 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG { 10:05:10 DEBUG "result": [ 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG { 10:05:10 DEBUG "data": { 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:05:10 DEBUG "result": [ 10:05:10 DEBUG }, 10:05:10 DEBUG { 10:05:10 DEBUG "value": [ 10:05:10 DEBUG "metric": { 10:05:10 DEBUG 10:05:10 DEBUG "instance": "10.142.0.11:9100" 10:05:10 DEBUG 10:05:10 DEBUG }, 10:05:10 DEBUG 10:05:10 DEBUG "value": [ 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:05:10 DEBUG 10:05:10 DEBUG }, 10:05:10 DEBUG 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG "instance": "10.142.0.51:9100" 10:05:10 DEBUG "15775342592" 10:05:10 DEBUG }, 10:05:10 DEBUG ] 10:05:10 DEBUG "value": [ 10:05:10 DEBUG } 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG ] 10:05:10 DEBUG "67804.73333333334" 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775510" 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG { 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "data": { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG { 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "instance": "10.142.0.11:9100" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "instance": "10.142.0.51:9100" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG "0.00033333333333350387" 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775510" 10:05:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775510" 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG { 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "data": { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG { 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "instance": "10.142.0.11:9100" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "instance": "10.142.0.51:9100" 10:05:10 DEBUG }, 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG "0.00036666666666652263" 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 INFO [http_cmd]: http status code OK 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG --- status code --- 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG --- http response --- 10:05:10 DEBUG { 10:05:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG ----- output ----- 10:05:10 DEBUG "data": { 10:05:10 DEBUG { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "status": "success", 10:05:10 DEBUG "result": [ 10:05:10 DEBUG "data": { 10:05:10 DEBUG { 10:05:10 DEBUG "resultType": "vector", 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "result": [ 10:05:10 DEBUG "pod": "ds-cts-0" 10:05:10 DEBUG { 10:05:10 DEBUG }, 10:05:10 DEBUG "metric": { 10:05:10 DEBUG "value": [ 10:05:10 DEBUG "pod": "ds-cts-0" 10:05:10 DEBUG 10:05:10 DEBUG }, 10:05:10 DEBUG 10:05:10 DEBUG "value": [ 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG [...] 10:05:10 DEBUG 10:05:10 DEBUG 10:05:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:05:10 DEBUG 10:05:10 DEBUG }, 10:05:10 DEBUG 10:05:10 DEBUG "value": [ 10:05:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG }, 10:05:10 DEBUG "0" 10:05:10 DEBUG "value": [ 10:05:10 DEBUG ] 10:05:10 DEBUG 1734775510, 10:05:10 DEBUG } 10:05:10 DEBUG "0.0005724704025267363" 10:05:10 DEBUG ] 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:10 DEBUG ] 10:05:10 DEBUG } 10:05:10 DEBUG } 10:05:53 INFO 10:05:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:05:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:05:53 INFO [loop_until]: OK (rc = 0) 10:05:53 DEBUG --- stdout --- 10:05:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 526Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 3m 842Mi 10:05:53 DEBUG --- stderr --- 10:05:53 DEBUG 10:05:58 INFO 10:05:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:05:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:05:58 INFO [loop_until]: OK (rc = 0) 10:05:58 DEBUG --- stdout --- 10:05:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 120m 0% 3173Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 106m 0% 19864Mi 69% 10:05:58 DEBUG --- stderr --- 10:05:58 DEBUG 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775569" 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:09 INFO [http_cmd]: http status code OK 10:06:09 DEBUG --- status code --- 10:06:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:09 DEBUG --- http response --- 10:06:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775570" 10:06:10 ERROR [http_cmd]: ERROR 10:06:10 ERROR --- status code --- 10:06:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:06:10 ERROR --- http response --- 10:06:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:06:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG { 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "data": { 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "result": [ 10:06:10 DEBUG { 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "pod": "ds-cts-0" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG [...] 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "732135424" 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG } 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG { 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "data": { 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "result": [ 10:06:10 DEBUG { 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG [...] 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "414.0243566663137" 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG } 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775570" 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG { 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG "data": { 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG { 10:06:10 DEBUG "result": [ 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG { 10:06:10 DEBUG "data": { 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "instance": "10.142.0.11:9100" 10:06:10 DEBUG "result": [ 10:06:10 DEBUG }, 10:06:10 DEBUG { 10:06:10 DEBUG "value": [ 10:06:10 DEBUG "metric": { 10:06:10 DEBUG 10:06:10 DEBUG "instance": "10.142.0.11:9100" 10:06:10 DEBUG 10:06:10 DEBUG }, 10:06:10 DEBUG 10:06:10 DEBUG "value": [ 10:06:10 DEBUG [...] 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG [...] 10:06:10 DEBUG "instance": "10.142.0.51:9100" 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG 10:06:10 DEBUG }, 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG 10:06:10 DEBUG "value": [ 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG "instance": "10.142.0.51:9100" 10:06:10 DEBUG "86908.26666666668" 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG }, 10:06:10 DEBUG ] 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG "value": [ 10:06:10 DEBUG } 10:06:10 DEBUG { 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG ] 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "25668.266666666666" 10:06:10 DEBUG } 10:06:10 DEBUG "data": { 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "result": [ 10:06:10 DEBUG { 10:06:10 DEBUG } 10:06:10 DEBUG "metric": { 10:06:10 DEBUG ] 10:06:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:06:10 DEBUG } 10:06:10 DEBUG }, 10:06:10 DEBUG } 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG [...] 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:06:10 DEBUG { 10:06:10 DEBUG }, 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "value": [ 10:06:10 DEBUG "data": { 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "0.04164648249387575" 10:06:10 DEBUG "result": [ 10:06:10 DEBUG ] 10:06:10 DEBUG { 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "instance": "10.142.0.11:9100" 10:06:10 DEBUG }, 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG } 10:06:10 DEBUG "value": [ 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG ] 10:06:10 DEBUG 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG [...] 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG 10:06:10 DEBUG { 10:06:10 DEBUG 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG 10:06:10 DEBUG "data": { 10:06:10 DEBUG "instance": "10.142.0.51:9100" 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG }, 10:06:10 DEBUG "result": [ 10:06:10 DEBUG "value": [ 10:06:10 DEBUG { 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "69304.1" 10:06:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:06:10 DEBUG ] 10:06:10 DEBUG }, 10:06:10 DEBUG } 10:06:10 DEBUG "value": [ 10:06:10 DEBUG ] 10:06:10 DEBUG 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG [...] 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "15780503552" 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG } 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775570" 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG { 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "data": { 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "result": [ 10:06:10 DEBUG { 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "instance": "10.142.0.11:9100" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG [...] 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG "instance": "10.142.0.51:9100" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "0" 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG } 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775570" 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG { 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "data": { 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "result": [ 10:06:10 DEBUG { 10:06:10 DEBUG "metric": { 10:06:10 DEBUG "instance": "10.142.0.11:9100" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG [...] 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG "instance": "10.142.0.51:9100" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "0.0003666666666636805" 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG } 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG { 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG "data": { 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG "result": [ 10:06:10 DEBUG { 10:06:10 INFO [http_cmd]: http status code OK 10:06:10 DEBUG "metric": { 10:06:10 DEBUG --- status code --- 10:06:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:06:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:06:10 DEBUG }, 10:06:10 DEBUG --- http response --- 10:06:10 DEBUG "value": [ 10:06:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:06:10 DEBUG 10:06:10 DEBUG ----- output ----- 10:06:10 DEBUG 10:06:10 DEBUG { 10:06:10 DEBUG 10:06:10 DEBUG "status": "success", 10:06:10 DEBUG [...] 10:06:10 DEBUG "data": { 10:06:10 DEBUG 10:06:10 DEBUG "resultType": "vector", 10:06:10 DEBUG 10:06:10 DEBUG "result": [ 10:06:10 DEBUG 10:06:10 DEBUG { 10:06:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:06:10 DEBUG "metric": { 10:06:10 DEBUG }, 10:06:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:06:10 DEBUG "value": [ 10:06:10 DEBUG }, 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG "0.00034694082035434203" 10:06:10 DEBUG 10:06:10 DEBUG ] 10:06:10 DEBUG 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG ] 10:06:10 DEBUG [...] 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG } 10:06:10 DEBUG 10:06:10 DEBUG 10:06:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:06:10 DEBUG }, 10:06:10 DEBUG "value": [ 10:06:10 DEBUG 1734775570, 10:06:10 DEBUG "0" 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG ] 10:06:10 DEBUG } 10:06:10 DEBUG } 10:06:53 INFO 10:06:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:06:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:06:53 INFO [loop_until]: OK (rc = 0) 10:06:53 DEBUG --- stdout --- 10:06:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 5m 13709Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:06:53 DEBUG --- stderr --- 10:06:53 DEBUG 10:06:58 INFO 10:06:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:06:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:06:58 INFO [loop_until]: OK (rc = 0) 10:06:58 DEBUG --- stdout --- 10:06:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 118m 0% 3180Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 102m 0% 19862Mi 69% 10:06:58 DEBUG --- stderr --- 10:06:58 DEBUG 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775629" 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775629" 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 INFO [http_cmd]: http status code OK 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG --- status code --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 DEBUG --- http response --- 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 ERROR [http_cmd]: ERROR 10:07:10 ERROR --- status code --- 10:07:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:07:10 ERROR --- http response --- 10:07:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:07:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG { 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "result": [ 10:07:10 DEBUG { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG "pod": "ds-cts-0" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG "732135424" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG { 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "result": [ 10:07:10 DEBUG { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG "pod": "ds-cts-0" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG "469.18552316575125" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775630" 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG { 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG { 10:07:10 DEBUG "data": { 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG "data": { 10:07:10 DEBUG "result": [ 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG { 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG "result": [ 10:07:10 DEBUG "metric": { 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG { 10:07:10 DEBUG { 10:07:10 DEBUG { 10:07:10 DEBUG "instance": "10.142.0.11:9100" 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG "data": { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG }, 10:07:10 DEBUG "result": [ 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "instance": "10.142.0.11:9100" 10:07:10 DEBUG "value": [ 10:07:10 DEBUG { 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG "result": [ 10:07:10 DEBUG }, 10:07:10 DEBUG 10:07:10 DEBUG "metric": { 10:07:10 DEBUG { 10:07:10 DEBUG { 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "metric": { 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG }, 10:07:10 DEBUG "data": { 10:07:10 DEBUG "instance": "10.142.0.11:9100" 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG "value": [ 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG }, 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "result": [ 10:07:10 DEBUG "value": [ 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG { 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "metric": { 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "instance": "10.142.0.51:9100" 10:07:10 DEBUG [...] 10:07:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG }, 10:07:10 DEBUG 10:07:10 DEBUG }, 10:07:10 DEBUG [...] 10:07:10 DEBUG "instance": "10.142.0.51:9100" 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG }, 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "value": [ 10:07:10 DEBUG "22664.533333333333" 10:07:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG ] 10:07:10 DEBUG }, 10:07:10 DEBUG 10:07:10 DEBUG "instance": "10.142.0.51:9100" 10:07:10 DEBUG "85343.79999999999" 10:07:10 DEBUG } 10:07:10 DEBUG "value": [ 10:07:10 DEBUG [...] 10:07:10 DEBUG }, 10:07:10 DEBUG ] 10:07:10 DEBUG ] 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG 10:07:10 DEBUG "value": [ 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 DEBUG "0.03603397359937593" 10:07:10 DEBUG 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG 10:07:10 DEBUG "67629.30000000002" 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG }, 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 DEBUG "value": [ 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG } 10:07:10 DEBUG "15779065856" 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775630" 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG { 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "result": [ 10:07:10 DEBUG { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG "instance": "10.142.0.11:9100" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "instance": "10.142.0.51:9100" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG "0.0003333333333330302" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775630" 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG { 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "result": [ 10:07:10 DEBUG { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG "instance": "10.142.0.11:9100" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "instance": "10.142.0.51:9100" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG "0.00039999999999906777" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:07:10 INFO [http_cmd]: http status code OK 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG --- status code --- 10:07:10 DEBUG { 10:07:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG --- http response --- 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "result": [ 10:07:10 DEBUG { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG "pod": "ds-cts-0" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG "0.0005000846430053629" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:07:10 DEBUG ----- output ----- 10:07:10 DEBUG { 10:07:10 DEBUG "status": "success", 10:07:10 DEBUG "data": { 10:07:10 DEBUG "resultType": "vector", 10:07:10 DEBUG "result": [ 10:07:10 DEBUG { 10:07:10 DEBUG "metric": { 10:07:10 DEBUG "pod": "ds-cts-0" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG [...] 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG 10:07:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:07:10 DEBUG }, 10:07:10 DEBUG "value": [ 10:07:10 DEBUG 1734775630, 10:07:10 DEBUG "0" 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG ] 10:07:10 DEBUG } 10:07:10 DEBUG } 10:07:53 INFO 10:07:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:07:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:07:53 INFO [loop_until]: OK (rc = 0) 10:07:53 DEBUG --- stdout --- 10:07:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:07:53 DEBUG --- stderr --- 10:07:53 DEBUG 10:07:58 INFO 10:07:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:07:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:07:58 INFO [loop_until]: OK (rc = 0) 10:07:58 DEBUG --- stdout --- 10:07:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 115m 0% 3213Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 101m 0% 19855Mi 69% 10:07:58 DEBUG --- stderr --- 10:07:58 DEBUG 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: http status code OK 10:08:09 DEBUG --- status code --- 10:08:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:09 DEBUG --- http response --- 10:08:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: http status code OK 10:08:09 DEBUG --- status code --- 10:08:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:09 INFO [http_cmd]: http status code OK 10:08:09 DEBUG --- http response --- 10:08:09 DEBUG --- status code --- 10:08:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:09 DEBUG --- http response --- 10:08:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:09 INFO [http_cmd]: http status code OK 10:08:09 DEBUG --- status code --- 10:08:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:09 DEBUG --- http response --- 10:08:09 INFO [http_cmd]: http status code OK 10:08:09 DEBUG --- status code --- 10:08:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:09 DEBUG --- http response --- 10:08:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775689" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 ERROR [http_cmd]: ERROR 10:08:10 ERROR --- status code --- 10:08:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 ERROR --- http response --- 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:08:10 DEBUG --- http response --- 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG { 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "data": { 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "result": [ 10:08:10 DEBUG { 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "pod": "ds-cts-0" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG [...] 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "732135424" 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG { 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "data": { 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "result": [ 10:08:10 DEBUG { 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG [...] 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "356.93161220631924" 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG { 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG "data": { 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG { 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG "result": [ 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG { 10:08:10 DEBUG "data": { 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "resultType": "vector", 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG "instance": "10.142.0.11:9100" 10:08:10 DEBUG "result": [ 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG { 10:08:10 DEBUG }, 10:08:10 DEBUG { 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "value": [ 10:08:10 DEBUG "metric": { 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG "data": { 10:08:10 DEBUG 10:08:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG }, 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG "result": [ 10:08:10 DEBUG 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG "value": [ 10:08:10 DEBUG { 10:08:10 DEBUG { 10:08:10 DEBUG [...] 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "metric": { 10:08:10 DEBUG 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG 10:08:10 DEBUG "data": { 10:08:10 DEBUG "instance": "10.142.0.11:9100" 10:08:10 DEBUG 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 DEBUG 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG }, 10:08:10 DEBUG 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG [...] 10:08:10 DEBUG "result": [ 10:08:10 DEBUG "value": [ 10:08:10 DEBUG "instance": "10.142.0.51:9100" 10:08:10 DEBUG { 10:08:10 DEBUG 10:08:10 DEBUG { 10:08:10 DEBUG 10:08:10 DEBUG }, 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG 10:08:10 DEBUG "metric": { 10:08:10 DEBUG 10:08:10 DEBUG "value": [ 10:08:10 DEBUG "data": { 10:08:10 DEBUG 10:08:10 DEBUG "instance": "10.142.0.11:9100" 10:08:10 DEBUG 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:08:10 DEBUG }, 10:08:10 DEBUG [...] 10:08:10 DEBUG "23483.733333333334" 10:08:10 DEBUG "result": [ 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG ] 10:08:10 DEBUG { 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG } 10:08:10 DEBUG "metric": { 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG ] 10:08:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:08:10 DEBUG "0.03763696288852221" 10:08:10 DEBUG 10:08:10 DEBUG "instance": "10.142.0.51:9100" 10:08:10 DEBUG } 10:08:10 DEBUG }, 10:08:10 DEBUG ] 10:08:10 DEBUG [...] 10:08:10 DEBUG }, 10:08:10 DEBUG } 10:08:10 DEBUG "value": [ 10:08:10 DEBUG } 10:08:10 DEBUG 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG ] 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG "instance": "10.142.0.51:9100" 10:08:10 DEBUG } 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG 10:08:10 DEBUG }, 10:08:10 DEBUG } 10:08:10 DEBUG "86984.13333333332" 10:08:10 DEBUG 10:08:10 DEBUG "value": [ 10:08:10 DEBUG ] 10:08:10 DEBUG [...] 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG } 10:08:10 DEBUG 10:08:10 DEBUG "69255.73333333334" 10:08:10 DEBUG ] 10:08:10 DEBUG 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:08:10 DEBUG ] 10:08:10 DEBUG }, 10:08:10 DEBUG } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775690" 10:08:10 DEBUG "value": [ 10:08:10 DEBUG } 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "15777710080" 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG { 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "data": { 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "result": [ 10:08:10 DEBUG { 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "instance": "10.142.0.11:9100" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG [...] 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG "instance": "10.142.0.51:9100" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "0.000666666666666534" 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775690" 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG { 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "data": { 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "result": [ 10:08:10 DEBUG { 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "instance": "10.142.0.11:9100" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG [...] 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG "instance": "10.142.0.51:9100" 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "0.0009333333333321056" 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 INFO [http_cmd]: http status code OK 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG --- status code --- 10:08:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:08:10 DEBUG --- http response --- 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:08:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG ----- output ----- 10:08:10 DEBUG { 10:08:10 DEBUG { 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "status": "success", 10:08:10 DEBUG "data": { 10:08:10 DEBUG "data": { 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "resultType": "vector", 10:08:10 DEBUG "result": [ 10:08:10 DEBUG "result": [ 10:08:10 DEBUG { 10:08:10 DEBUG { 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "metric": { 10:08:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:08:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:08:10 DEBUG }, 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG [...] 10:08:10 DEBUG [...] 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG 10:08:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:08:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:08:10 DEBUG }, 10:08:10 DEBUG }, 10:08:10 DEBUG "value": [ 10:08:10 DEBUG "value": [ 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG 1734775690, 10:08:10 DEBUG "0.0003553340179111895" 10:08:10 DEBUG "0" 10:08:10 DEBUG ] 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 DEBUG ] 10:08:10 DEBUG ] 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:10 DEBUG } 10:08:53 INFO 10:08:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:08:53 INFO [loop_until]: OK (rc = 0) 10:08:53 DEBUG --- stdout --- 10:08:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:08:53 DEBUG --- stderr --- 10:08:53 DEBUG 10:08:58 INFO 10:08:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:08:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:08:58 INFO [loop_until]: OK (rc = 0) 10:08:58 DEBUG --- stdout --- 10:08:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 116m 0% 3214Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 101m 0% 19861Mi 69% 10:08:58 DEBUG --- stderr --- 10:08:58 DEBUG 10:09:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775749" 10:09:09 INFO [http_cmd]: http status code OK 10:09:09 DEBUG --- status code --- 10:09:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:09 DEBUG --- http response --- 10:09:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775749" 10:09:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775749" 10:09:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775749" 10:09:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775749" 10:09:09 INFO [http_cmd]: http status code OK 10:09:09 DEBUG --- status code --- 10:09:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:09 DEBUG --- http response --- 10:09:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:09 INFO [http_cmd]: http status code OK 10:09:09 DEBUG --- status code --- 10:09:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:09 DEBUG --- http response --- 10:09:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:09 INFO [http_cmd]: http status code OK 10:09:09 INFO [http_cmd]: http status code OK 10:09:09 DEBUG --- status code --- 10:09:09 DEBUG --- status code --- 10:09:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:09 DEBUG --- http response --- 10:09:09 DEBUG --- http response --- 10:09:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775750" 10:09:10 ERROR [http_cmd]: ERROR 10:09:10 ERROR --- status code --- 10:09:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:09:10 ERROR --- http response --- 10:09:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:09:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG --- status code --- 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775750" 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG { 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "data": { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "result": [ 10:09:10 DEBUG { 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "pod": "ds-cts-0" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG "732135424" 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG { 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "data": { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "result": [ 10:09:10 DEBUG { 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "pod": "ds-cts-0" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG "574.7270636400018" 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775750" 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG { 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "data": { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "result": [ 10:09:10 DEBUG { 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "instance": "10.142.0.11:9100" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG 10:09:10 DEBUG { 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG [...] 10:09:10 DEBUG "data": { 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 DEBUG 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775750" 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG "result": [ 10:09:10 DEBUG { 10:09:10 DEBUG 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG { 10:09:10 DEBUG { 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "instance": "10.142.0.51:9100" 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "data": { 10:09:10 DEBUG }, 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG "data": { 10:09:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "value": [ 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG }, 10:09:10 DEBUG "result": [ 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG "result": [ 10:09:10 DEBUG "value": [ 10:09:10 DEBUG { 10:09:10 DEBUG "85357.83333333334" 10:09:10 DEBUG { 10:09:10 DEBUG { 10:09:10 DEBUG 10:09:10 DEBUG "metric": { 10:09:10 DEBUG ] 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "metric": { 10:09:10 DEBUG 10:09:10 DEBUG "instance": "10.142.0.11:9100" 10:09:10 DEBUG } 10:09:10 DEBUG "data": { 10:09:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:09:10 DEBUG 10:09:10 DEBUG }, 10:09:10 DEBUG ] 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG }, 10:09:10 DEBUG [...] 10:09:10 DEBUG "value": [ 10:09:10 DEBUG } 10:09:10 DEBUG "result": [ 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG } 10:09:10 DEBUG { 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG "metric": { 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:09:10 DEBUG "instance": "10.142.0.11:9100" 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG }, 10:09:10 DEBUG }, 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG "value": [ 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG "instance": "10.142.0.51:9100" 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG }, 10:09:10 DEBUG "15776813056" 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "value": [ 10:09:10 DEBUG ] 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775750" 10:09:10 DEBUG 10:09:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG } 10:09:10 DEBUG [...] 10:09:10 DEBUG }, 10:09:10 DEBUG "67680.90000000002" 10:09:10 DEBUG ] 10:09:10 DEBUG 10:09:10 DEBUG "value": [ 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 DEBUG 10:09:10 DEBUG "0.040708151812489754" 10:09:10 DEBUG ] 10:09:10 DEBUG "instance": "10.142.0.51:9100" 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG }, 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 DEBUG "value": [ 10:09:10 DEBUG ] 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG } 10:09:10 DEBUG "30037.333333333336" 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG { 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "data": { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "result": [ 10:09:10 DEBUG { 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "instance": "10.142.0.11:9100" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "instance": "10.142.0.51:9100" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG "0.00033333333333350387" 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775750" 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG { 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "data": { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "result": [ 10:09:10 DEBUG { 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "instance": "10.142.0.11:9100" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "instance": "10.142.0.51:9100" 10:09:10 DEBUG }, 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG "0.0004666666666698423" 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 INFO [http_cmd]: http status code OK 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG --- status code --- 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG --- http response --- 10:09:10 DEBUG { 10:09:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG ----- output ----- 10:09:10 DEBUG "data": { 10:09:10 DEBUG { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "status": "success", 10:09:10 DEBUG "result": [ 10:09:10 DEBUG "data": { 10:09:10 DEBUG { 10:09:10 DEBUG "resultType": "vector", 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "result": [ 10:09:10 DEBUG "pod": "ds-cts-0" 10:09:10 DEBUG { 10:09:10 DEBUG }, 10:09:10 DEBUG "metric": { 10:09:10 DEBUG "value": [ 10:09:10 DEBUG "pod": "ds-cts-0" 10:09:10 DEBUG 10:09:10 DEBUG }, 10:09:10 DEBUG 10:09:10 DEBUG "value": [ 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG [...] 10:09:10 DEBUG 10:09:10 DEBUG 10:09:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:09:10 DEBUG 10:09:10 DEBUG }, 10:09:10 DEBUG 10:09:10 DEBUG "value": [ 10:09:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG }, 10:09:10 DEBUG "0" 10:09:10 DEBUG "value": [ 10:09:10 DEBUG ] 10:09:10 DEBUG 1734775750, 10:09:10 DEBUG } 10:09:10 DEBUG "0.0006940995309697973" 10:09:10 DEBUG ] 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:10 DEBUG ] 10:09:10 DEBUG } 10:09:10 DEBUG } 10:09:53 INFO 10:09:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:09:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:09:53 INFO [loop_until]: OK (rc = 0) 10:09:53 DEBUG --- stdout --- 10:09:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:09:53 DEBUG --- stderr --- 10:09:53 DEBUG 10:09:58 INFO 10:09:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:09:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:09:58 INFO [loop_until]: OK (rc = 0) 10:09:58 DEBUG --- stdout --- 10:09:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 118m 0% 3170Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 100m 0% 19861Mi 69% 10:09:58 DEBUG --- stderr --- 10:09:58 DEBUG 10:10:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775809" 10:10:09 INFO [http_cmd]: http status code OK 10:10:09 DEBUG --- status code --- 10:10:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:09 DEBUG --- http response --- 10:10:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775809" 10:10:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775809" 10:10:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775809" 10:10:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775809" 10:10:09 INFO [http_cmd]: http status code OK 10:10:09 DEBUG --- status code --- 10:10:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:09 DEBUG --- http response --- 10:10:09 INFO [http_cmd]: http status code OK 10:10:09 INFO [http_cmd]: http status code OK 10:10:09 INFO [http_cmd]: http status code OK 10:10:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:09 DEBUG --- status code --- 10:10:09 DEBUG --- status code --- 10:10:09 DEBUG --- status code --- 10:10:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:09 DEBUG --- http response --- 10:10:09 DEBUG --- http response --- 10:10:09 DEBUG --- http response --- 10:10:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 ERROR [http_cmd]: ERROR 10:10:10 ERROR --- status code --- 10:10:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:10:10 ERROR --- http response --- 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775810" 10:10:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG { 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "data": { 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "result": [ 10:10:10 DEBUG { 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "pod": "ds-cts-0" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG [...] 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "732135424" 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG { 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "data": { 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "result": [ 10:10:10 DEBUG { 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "pod": "ds-cts-0" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG [...] 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "253.85661574580817" 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775810" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG { 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "data": { 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "result": [ 10:10:10 DEBUG { 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "instance": "10.142.0.11:9100" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG [...] 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "instance": "10.142.0.51:9100" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "85168.30000000002" 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775810" 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG { 10:10:10 DEBUG { 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG "data": { 10:10:10 DEBUG "data": { 10:10:10 DEBUG { 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "result": [ 10:10:10 DEBUG "result": [ 10:10:10 DEBUG "data": { 10:10:10 DEBUG { 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG { 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "metric": { 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "result": [ 10:10:10 DEBUG "instance": "10.142.0.11:9100" 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:10:10 DEBUG { 10:10:10 DEBUG }, 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG }, 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "value": [ 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG "value": [ 10:10:10 DEBUG "instance": "10.142.0.11:9100" 10:10:10 DEBUG 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG 10:10:10 DEBUG }, 10:10:10 DEBUG 10:10:10 DEBUG { 10:10:10 DEBUG { 10:10:10 DEBUG 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG [...] 10:10:10 DEBUG "data": { 10:10:10 DEBUG "data": { 10:10:10 DEBUG [...] 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "result": [ 10:10:10 DEBUG "result": [ 10:10:10 DEBUG 10:10:10 DEBUG [...] 10:10:10 DEBUG 10:10:10 DEBUG { 10:10:10 DEBUG { 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "instance": "10.142.0.51:9100" 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:10:10 DEBUG 10:10:10 DEBUG }, 10:10:10 DEBUG "instance": "10.142.0.11:9100" 10:10:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:10:10 DEBUG }, 10:10:10 DEBUG 10:10:10 DEBUG "value": [ 10:10:10 DEBUG }, 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG "instance": "10.142.0.51:9100" 10:10:10 DEBUG 1734775810, 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775810" 10:10:10 DEBUG "value": [ 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG }, 10:10:10 DEBUG "67421.4" 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG 10:10:10 DEBUG "value": [ 10:10:10 DEBUG ] 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG 10:10:10 DEBUG "15780118528" 10:10:10 DEBUG 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG } 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG 10:10:10 DEBUG ] 10:10:10 DEBUG [...] 10:10:10 DEBUG "0.00033333333333350387" 10:10:10 DEBUG ] 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG [...] 10:10:10 DEBUG } 10:10:10 DEBUG 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG 10:10:10 DEBUG ] 10:10:10 DEBUG 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG 10:10:10 DEBUG } 10:10:10 DEBUG 10:10:10 DEBUG ] 10:10:10 DEBUG { 10:10:10 DEBUG 10:10:10 DEBUG } 10:10:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:10:10 DEBUG } 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "instance": "10.142.0.51:9100" 10:10:10 DEBUG }, 10:10:10 DEBUG } 10:10:10 DEBUG "data": { 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "result": [ 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "0.04171991173029516" 10:10:10 DEBUG { 10:10:10 DEBUG "17612.8" 10:10:10 DEBUG ] 10:10:10 DEBUG "metric": { 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG "instance": "10.142.0.11:9100" 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG } 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG ] 10:10:10 INFO [http_cmd]: http status code OK 10:10:10 DEBUG 10:10:10 DEBUG ] 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG } 10:10:10 DEBUG --- status code --- 10:10:10 DEBUG 10:10:10 DEBUG } 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG } 10:10:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:10:10 DEBUG [...] 10:10:10 DEBUG } 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG --- http response --- 10:10:10 DEBUG 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:10:10 DEBUG 10:10:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG 10:10:10 DEBUG ----- output ----- 10:10:10 DEBUG { 10:10:10 DEBUG "instance": "10.142.0.51:9100" 10:10:10 DEBUG { 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG }, 10:10:10 DEBUG "status": "success", 10:10:10 DEBUG "data": { 10:10:10 DEBUG "value": [ 10:10:10 DEBUG "data": { 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "resultType": "vector", 10:10:10 DEBUG "result": [ 10:10:10 DEBUG "0.00033333333333397756" 10:10:10 DEBUG "result": [ 10:10:10 DEBUG { 10:10:10 DEBUG ] 10:10:10 DEBUG { 10:10:10 DEBUG "metric": { 10:10:10 DEBUG } 10:10:10 DEBUG "metric": { 10:10:10 DEBUG "pod": "ds-cts-0" 10:10:10 DEBUG ] 10:10:10 DEBUG "pod": "ds-cts-0" 10:10:10 DEBUG }, 10:10:10 DEBUG } 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG } 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG [...] 10:10:10 DEBUG [...] 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG 10:10:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:10:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:10:10 DEBUG }, 10:10:10 DEBUG }, 10:10:10 DEBUG "value": [ 10:10:10 DEBUG "value": [ 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG 1734775810, 10:10:10 DEBUG "0.0004031275285867032" 10:10:10 DEBUG "0" 10:10:10 DEBUG ] 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 DEBUG ] 10:10:10 DEBUG ] 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:10 DEBUG } 10:10:53 INFO 10:10:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:10:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:10:53 INFO [loop_until]: OK (rc = 0) 10:10:53 DEBUG --- stdout --- 10:10:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:10:53 DEBUG --- stderr --- 10:10:53 DEBUG 10:10:58 INFO 10:10:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:10:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:10:58 INFO [loop_until]: OK (rc = 0) 10:10:58 DEBUG --- stdout --- 10:10:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 111m 0% 3190Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 96m 0% 19863Mi 69% 10:10:58 DEBUG --- stderr --- 10:10:58 DEBUG 10:11:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775869" 10:11:09 INFO [http_cmd]: http status code OK 10:11:09 DEBUG --- status code --- 10:11:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:09 DEBUG --- http response --- 10:11:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775869" 10:11:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775869" 10:11:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775869" 10:11:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775869" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG --- status code --- 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- status code --- 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 ERROR [http_cmd]: ERROR 10:11:10 ERROR --- status code --- 10:11:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 ERROR --- http response --- 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775870" 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- status code --- 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG { 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG "data": { 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "result": [ 10:11:10 DEBUG { 10:11:10 DEBUG "metric": { 10:11:10 DEBUG "pod": "ds-cts-0" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG [...] 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG "732135424" 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG { 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG "data": { 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "result": [ 10:11:10 DEBUG { 10:11:10 DEBUG "metric": { 10:11:10 DEBUG "pod": "ds-cts-0" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG [...] 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG "850.2470754252387" 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775870" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG { 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG "data": { 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "result": [ 10:11:10 DEBUG { 10:11:10 DEBUG "metric": { 10:11:10 DEBUG "instance": "10.142.0.11:9100" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG [...] 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "instance": "10.142.0.51:9100" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG "87245.8" 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775870" 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG --- status code --- 10:11:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775870" 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG { 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG { 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG "data": { 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG "data": { 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG { 10:11:10 DEBUG "result": [ 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG "result": [ 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG { 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG { 10:11:10 DEBUG "data": { 10:11:10 DEBUG "metric": { 10:11:10 DEBUG { 10:11:10 DEBUG { 10:11:10 DEBUG "metric": { 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG "status": "success", 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG "instance": "10.142.0.11:9100" 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG "result": [ 10:11:10 DEBUG }, 10:11:10 DEBUG "data": { 10:11:10 DEBUG "data": { 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG }, 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG { 10:11:10 DEBUG "value": [ 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG "metric": { 10:11:10 DEBUG 10:11:10 DEBUG "result": [ 10:11:10 DEBUG "result": [ 10:11:10 DEBUG "value": [ 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG "instance": "10.142.0.11:9100" 10:11:10 DEBUG 10:11:10 DEBUG { 10:11:10 DEBUG { 10:11:10 DEBUG 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:11:10 DEBUG }, 10:11:10 DEBUG 10:11:10 DEBUG "metric": { 10:11:10 DEBUG "metric": { 10:11:10 DEBUG 10:11:10 DEBUG { 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG "value": [ 10:11:10 DEBUG [...] 10:11:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:11:10 DEBUG "instance": "10.142.0.11:9100" 10:11:10 DEBUG 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG { 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG }, 10:11:10 DEBUG }, 10:11:10 DEBUG [...] 10:11:10 DEBUG "data": { 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "value": [ 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "data": { 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "result": [ 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG [...] 10:11:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG { 10:11:10 DEBUG "result": [ 10:11:10 DEBUG 10:11:10 DEBUG }, 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "instance": "10.142.0.51:9100" 10:11:10 DEBUG "metric": { 10:11:10 INFO [http_cmd]: http status code OK 10:11:10 DEBUG { 10:11:10 DEBUG 10:11:10 DEBUG "value": [ 10:11:10 DEBUG [...] 10:11:10 DEBUG [...] 10:11:10 DEBUG }, 10:11:10 DEBUG "instance": "10.142.0.11:9100" 10:11:10 DEBUG --- status code --- 10:11:10 DEBUG "metric": { 10:11:10 DEBUG 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG "pod": "ds-cts-0" 10:11:10 DEBUG "instance": "10.142.0.51:9100" 10:11:10 DEBUG "15781072896" 10:11:10 DEBUG 10:11:10 DEBUG "value": [ 10:11:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:11:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:11:10 DEBUG 10:11:10 DEBUG }, 10:11:10 DEBUG }, 10:11:10 DEBUG ] 10:11:10 DEBUG 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG }, 10:11:10 DEBUG --- http response --- 10:11:10 DEBUG 10:11:10 DEBUG "value": [ 10:11:10 DEBUG "value": [ 10:11:10 DEBUG } 10:11:10 DEBUG 10:11:10 DEBUG "71842.63333333335" 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 10:11:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:11:10 DEBUG 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG ] 10:11:10 DEBUG "instance": "10.142.0.51:9100" 10:11:10 DEBUG ] 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG [...] 10:11:10 DEBUG ----- output ----- 10:11:10 DEBUG 10:11:10 DEBUG "0.0006666666666670077" 10:11:10 DEBUG } 10:11:10 DEBUG }, 10:11:10 DEBUG } 10:11:10 DEBUG "0.04271175394819044" 10:11:10 DEBUG 10:11:10 DEBUG { 10:11:10 DEBUG 10:11:10 DEBUG [...] 10:11:10 DEBUG } 10:11:10 DEBUG "value": [ 10:11:10 DEBUG ] 10:11:10 DEBUG ] 10:11:10 DEBUG 10:11:10 DEBUG "status": "success", 10:11:10 DEBUG ] 10:11:10 DEBUG 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 DEBUG 10:11:10 DEBUG "data": { 10:11:10 DEBUG } 10:11:10 DEBUG 10:11:10 DEBUG "17066.666666666668" 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG "instance": "10.142.0.51:9100" 10:11:10 DEBUG "resultType": "vector", 10:11:10 DEBUG "result": [ 10:11:10 DEBUG 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG { 10:11:10 DEBUG "0.00046666666666700016" 10:11:10 DEBUG }, 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG "metric": { 10:11:10 DEBUG ] 10:11:10 DEBUG "value": [ 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 DEBUG "pod": "ds-cts-0" 10:11:10 DEBUG } 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG } 10:11:10 DEBUG }, 10:11:10 DEBUG ] 10:11:10 DEBUG "0" 10:11:10 DEBUG "value": [ 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:10 DEBUG 10:11:10 DEBUG ] 10:11:10 DEBUG 10:11:10 DEBUG } 10:11:10 DEBUG [...] 10:11:10 DEBUG } 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG 10:11:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:11:10 DEBUG }, 10:11:10 DEBUG "value": [ 10:11:10 DEBUG 1734775870, 10:11:10 DEBUG "0.0011357724679465662" 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG ] 10:11:10 DEBUG } 10:11:10 DEBUG } 10:11:53 INFO 10:11:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:11:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:11:53 INFO [loop_until]: OK (rc = 0) 10:11:53 DEBUG --- stdout --- 10:11:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:11:53 DEBUG --- stderr --- 10:11:53 DEBUG 10:11:58 INFO 10:11:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:11:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:11:58 INFO [loop_until]: OK (rc = 0) 10:11:58 DEBUG --- stdout --- 10:11:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 109m 0% 3122Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 100m 0% 19863Mi 69% 10:11:58 DEBUG --- stderr --- 10:11:58 DEBUG 10:12:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775929" 10:12:09 INFO [http_cmd]: http status code OK 10:12:09 DEBUG --- status code --- 10:12:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:09 DEBUG --- http response --- 10:12:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 ERROR [http_cmd]: ERROR 10:12:10 ERROR --- status code --- 10:12:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:12:10 ERROR --- http response --- 10:12:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:12:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775930" 10:12:10 DEBUG --- status code --- 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG { 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG "data": { 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG "result": [ 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "metric": { 10:12:10 DEBUG "data": { 10:12:10 DEBUG "pod": "ds-cts-0" 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG }, 10:12:10 DEBUG "result": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG { 10:12:10 DEBUG 10:12:10 DEBUG "metric": { 10:12:10 DEBUG 10:12:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:12:10 DEBUG 10:12:10 DEBUG }, 10:12:10 DEBUG [...] 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:12:10 DEBUG [...] 10:12:10 DEBUG }, 10:12:10 DEBUG 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG 10:12:10 DEBUG "732135424" 10:12:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:12:10 DEBUG ] 10:12:10 DEBUG }, 10:12:10 DEBUG } 10:12:10 DEBUG "value": [ 10:12:10 DEBUG ] 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG } 10:12:10 DEBUG "535.8626328699918" 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775930" 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG { 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "data": { 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG "result": [ 10:12:10 DEBUG { 10:12:10 DEBUG "metric": { 10:12:10 DEBUG "instance": "10.142.0.11:9100" 10:12:10 DEBUG }, 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG [...] 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "instance": "10.142.0.51:9100" 10:12:10 DEBUG }, 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG "86024.06666666668" 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775930" 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG --- http response --- 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG { 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG "status": "success", 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG "data": { 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG "data": { 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG "result": [ 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "result": [ 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG { 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG "data": { 10:12:10 DEBUG "data": { 10:12:10 DEBUG { 10:12:10 DEBUG "metric": { 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG "metric": { 10:12:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG "result": [ 10:12:10 INFO [http_cmd]: http status code OK 10:12:10 DEBUG "result": [ 10:12:10 DEBUG "instance": "10.142.0.11:9100" 10:12:10 DEBUG }, 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG --- status code --- 10:12:10 DEBUG { 10:12:10 DEBUG }, 10:12:10 DEBUG "value": [ 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG "metric": { 10:12:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:12:10 DEBUG "metric": { 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG "data": { 10:12:10 DEBUG "data": { 10:12:10 DEBUG "data": { 10:12:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:12:10 DEBUG --- http response --- 10:12:10 DEBUG "instance": "10.142.0.11:9100" 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG }, 10:12:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:12:10 DEBUG }, 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "result": [ 10:12:10 DEBUG "result": [ 10:12:10 DEBUG "result": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG ----- output ----- 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG [...] 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG { 10:12:10 DEBUG 10:12:10 DEBUG { 10:12:10 DEBUG 10:12:10 DEBUG [...] 10:12:10 DEBUG 10:12:10 DEBUG "metric": { 10:12:10 DEBUG "metric": { 10:12:10 DEBUG "metric": { 10:12:10 DEBUG 10:12:10 DEBUG "status": "success", 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:12:10 DEBUG "instance": "10.142.0.11:9100" 10:12:10 DEBUG "instance": "10.142.0.11:9100" 10:12:10 DEBUG 10:12:10 DEBUG "data": { 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG }, 10:12:10 DEBUG }, 10:12:10 DEBUG }, 10:12:10 DEBUG [...] 10:12:10 DEBUG "resultType": "vector", 10:12:10 DEBUG [...] 10:12:10 DEBUG 10:12:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:12:10 DEBUG "value": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG "result": [ 10:12:10 DEBUG 10:12:10 DEBUG "instance": "10.142.0.51:9100" 10:12:10 DEBUG }, 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG { 10:12:10 DEBUG 10:12:10 DEBUG }, 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "metric": { 10:12:10 DEBUG 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:12:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:12:10 DEBUG "instance": "10.142.0.51:9100" 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG "0.05104966622611648" 10:12:10 DEBUG [...] 10:12:10 DEBUG [...] 10:12:10 DEBUG [...] 10:12:10 DEBUG }, 10:12:10 DEBUG }, 10:12:10 DEBUG }, 10:12:10 DEBUG "67786.66666666667" 10:12:10 DEBUG ] 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "value": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG 10:12:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:12:10 DEBUG }, 10:12:10 DEBUG "value": [ 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG "0" 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG 10:12:10 DEBUG 10:12:10 DEBUG "15777030144" 10:12:10 DEBUG 10:12:10 DEBUG "0.0006666666666670077" 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 DEBUG "instance": "10.142.0.51:9100" 10:12:10 DEBUG "instance": "10.142.0.51:9100" 10:12:10 DEBUG ] 10:12:10 DEBUG 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 DEBUG }, 10:12:10 DEBUG }, 10:12:10 DEBUG } 10:12:10 DEBUG [...] 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG "value": [ 10:12:10 DEBUG "value": [ 10:12:10 DEBUG ] 10:12:10 DEBUG 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG } 10:12:10 DEBUG 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 DEBUG "0.0012000000000028876" 10:12:10 DEBUG "20343.466666666667" 10:12:10 DEBUG } 10:12:10 DEBUG 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG ] 10:12:10 DEBUG "pod": "ds-idrepo-0" 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 DEBUG }, 10:12:10 DEBUG ] 10:12:10 DEBUG ] 10:12:10 DEBUG "value": [ 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 DEBUG 1734775930, 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:10 DEBUG "0" 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG ] 10:12:10 DEBUG } 10:12:10 DEBUG } 10:12:53 INFO 10:12:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:12:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:12:53 INFO [loop_until]: OK (rc = 0) 10:12:53 DEBUG --- stdout --- 10:12:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:12:53 DEBUG --- stderr --- 10:12:53 DEBUG 10:12:58 INFO 10:12:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:12:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:12:58 INFO [loop_until]: OK (rc = 0) 10:12:58 DEBUG --- stdout --- 10:12:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 106m 0% 3158Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19859Mi 69% 10:12:58 DEBUG --- stderr --- 10:12:58 DEBUG 10:13:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775989" 10:13:09 INFO [http_cmd]: http status code OK 10:13:09 DEBUG --- status code --- 10:13:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:09 DEBUG --- http response --- 10:13:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734775990" 10:13:10 ERROR [http_cmd]: ERROR 10:13:10 ERROR --- status code --- 10:13:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:13:10 ERROR --- http response --- 10:13:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:13:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG "data": { 10:13:10 DEBUG { 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "result": [ 10:13:10 DEBUG "data": { 10:13:10 DEBUG { 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "metric": { 10:13:10 DEBUG "result": [ 10:13:10 DEBUG "pod": "ds-cts-0" 10:13:10 DEBUG { 10:13:10 DEBUG }, 10:13:10 DEBUG "metric": { 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "pod": "ds-cts-0" 10:13:10 DEBUG 10:13:10 DEBUG }, 10:13:10 DEBUG 10:13:10 DEBUG "value": [ 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG [...] 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG [...] 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:13:10 DEBUG 10:13:10 DEBUG }, 10:13:10 DEBUG 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG }, 10:13:10 DEBUG "732135424" 10:13:10 DEBUG "value": [ 10:13:10 DEBUG ] 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG } 10:13:10 DEBUG "223.88018911754256" 10:13:10 DEBUG ] 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG { 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "data": { 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "result": [ 10:13:10 DEBUG { 10:13:10 DEBUG "metric": { 10:13:10 DEBUG "instance": "10.142.0.11:9100" 10:13:10 DEBUG }, 10:13:10 DEBUG "value": [ 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG [...] 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "instance": "10.142.0.51:9100" 10:13:10 DEBUG }, 10:13:10 DEBUG "value": [ 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG "87784.20000000003" 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734775990" 10:13:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734775990" 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG { 10:13:10 DEBUG "status": "success", 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG "data": { 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG "result": [ 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG { 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:13:10 DEBUG "metric": { 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG "instance": "10.142.0.11:9100" 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG { 10:13:10 DEBUG }, 10:13:10 DEBUG { 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "value": [ 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG "data": { 10:13:10 DEBUG 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG "data": { 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG "result": [ 10:13:10 DEBUG 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG "result": [ 10:13:10 DEBUG { 10:13:10 DEBUG { 10:13:10 DEBUG { 10:13:10 DEBUG [...] 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:13:10 DEBUG { 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "metric": { 10:13:10 DEBUG 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG "metric": { 10:13:10 DEBUG "data": { 10:13:10 DEBUG "data": { 10:13:10 DEBUG "pod": "ds-cts-0" 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "instance": "10.142.0.51:9100" 10:13:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG }, 10:13:10 DEBUG { 10:13:10 DEBUG { 10:13:10 DEBUG }, 10:13:10 INFO [http_cmd]: http status code OK 10:13:10 DEBUG }, 10:13:10 DEBUG "result": [ 10:13:10 DEBUG "result": [ 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG "value": [ 10:13:10 DEBUG --- status code --- 10:13:10 DEBUG "value": [ 10:13:10 DEBUG { 10:13:10 DEBUG { 10:13:10 DEBUG 10:13:10 DEBUG "data": { 10:13:10 DEBUG "data": { 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:13:10 DEBUG 10:13:10 DEBUG "metric": { 10:13:10 DEBUG "metric": { 10:13:10 DEBUG 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "69570.86666666668" 10:13:10 DEBUG --- http response --- 10:13:10 DEBUG 10:13:10 DEBUG "instance": "10.142.0.11:9100" 10:13:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:13:10 DEBUG 10:13:10 DEBUG "result": [ 10:13:10 DEBUG "result": [ 10:13:10 DEBUG ] 10:13:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:13:10 DEBUG 10:13:10 DEBUG }, 10:13:10 DEBUG }, 10:13:10 DEBUG [...] 10:13:10 DEBUG { 10:13:10 DEBUG { 10:13:10 DEBUG } 10:13:10 DEBUG ----- output ----- 10:13:10 DEBUG [...] 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "value": [ 10:13:10 DEBUG 10:13:10 DEBUG "metric": { 10:13:10 DEBUG "metric": { 10:13:10 DEBUG ] 10:13:10 DEBUG { 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "instance": "10.142.0.11:9100" 10:13:10 DEBUG "instance": "10.142.0.11:9100" 10:13:10 DEBUG } 10:13:10 DEBUG "status": "success", 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG }, 10:13:10 DEBUG }, 10:13:10 DEBUG } 10:13:10 DEBUG "data": { 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "resultType": "vector", 10:13:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:13:10 DEBUG [...] 10:13:10 DEBUG [...] 10:13:10 DEBUG }, 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "result": [ 10:13:10 DEBUG }, 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "value": [ 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG { 10:13:10 DEBUG "value": [ 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "metric": { 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "0" 10:13:10 DEBUG [...] 10:13:10 DEBUG [...] 10:13:10 DEBUG "pod": "ds-cts-0" 10:13:10 DEBUG "0.042502588701667496" 10:13:10 DEBUG "instance": "10.142.0.51:9100" 10:13:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:13:10 DEBUG ] 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG }, 10:13:10 DEBUG ] 10:13:10 DEBUG }, 10:13:10 DEBUG }, 10:13:10 DEBUG } 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "value": [ 10:13:10 DEBUG } 10:13:10 DEBUG "value": [ 10:13:10 DEBUG "value": [ 10:13:10 DEBUG ] 10:13:10 DEBUG 10:13:10 DEBUG "instance": "10.142.0.51:9100" 10:13:10 DEBUG 10:13:10 DEBUG ] 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG }, 10:13:10 DEBUG 10:13:10 DEBUG 10:13:10 DEBUG "0.00033333333333350387" 10:13:10 DEBUG "15777521664" 10:13:10 DEBUG 10:13:10 DEBUG "value": [ 10:13:10 DEBUG } 10:13:10 DEBUG [...] 10:13:10 DEBUG ] 10:13:10 DEBUG ] 10:13:10 DEBUG "instance": "10.142.0.51:9100" 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG } 10:13:10 DEBUG 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG }, 10:13:10 DEBUG "31266.13333333333" 10:13:10 DEBUG 10:13:10 DEBUG ] 10:13:10 DEBUG ] 10:13:10 DEBUG "value": [ 10:13:10 DEBUG ] 10:13:10 DEBUG 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG } 10:13:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG "0.00046666666666605273" 10:13:10 DEBUG ] 10:13:10 DEBUG }, 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG "value": [ 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG 1734775990, 10:13:10 DEBUG ] 10:13:10 DEBUG "0.0005688557295509529" 10:13:10 DEBUG } 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:10 DEBUG ] 10:13:10 DEBUG } 10:13:10 DEBUG } 10:13:53 INFO 10:13:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:13:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:13:53 INFO [loop_until]: OK (rc = 0) 10:13:53 DEBUG --- stdout --- 10:13:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:13:53 DEBUG --- stderr --- 10:13:53 DEBUG 10:13:58 INFO 10:13:58 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:13:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:13:59 INFO [loop_until]: OK (rc = 0) 10:13:59 DEBUG --- stdout --- 10:13:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3172Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 104m 0% 19860Mi 69% 10:13:59 DEBUG --- stderr --- 10:13:59 DEBUG 10:14:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776049" 10:14:09 INFO [http_cmd]: http status code OK 10:14:09 DEBUG --- status code --- 10:14:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:09 DEBUG --- http response --- 10:14:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG --- status code --- 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776050" 10:14:10 ERROR [http_cmd]: ERROR 10:14:10 ERROR --- status code --- 10:14:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:14:10 ERROR --- http response --- 10:14:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:14:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG { 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG "data": { 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG "result": [ 10:14:10 DEBUG { 10:14:10 DEBUG "metric": { 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:10 DEBUG "pod": "ds-cts-0" 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG }, 10:14:10 DEBUG { 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG 10:14:10 DEBUG "data": { 10:14:10 DEBUG 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG 10:14:10 DEBUG "result": [ 10:14:10 DEBUG [...] 10:14:10 DEBUG { 10:14:10 DEBUG 10:14:10 DEBUG "metric": { 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:14:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:14:10 DEBUG }, 10:14:10 DEBUG }, 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG 10:14:10 DEBUG "732135424" 10:14:10 DEBUG 10:14:10 DEBUG ] 10:14:10 DEBUG [...] 10:14:10 DEBUG } 10:14:10 DEBUG 10:14:10 DEBUG ] 10:14:10 DEBUG 10:14:10 DEBUG } 10:14:10 DEBUG 10:14:10 DEBUG } 10:14:10 DEBUG "pod": "ds-idrepo-0" 10:14:10 DEBUG }, 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG "588.0951124965032" 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG { 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG "data": { 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG "result": [ 10:14:10 DEBUG { 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "instance": "10.142.0.11:9100" 10:14:10 DEBUG }, 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG [...] 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "instance": "10.142.0.51:9100" 10:14:10 DEBUG }, 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG "84805.26666666666" 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776050" 10:14:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776050" 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG { 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG "data": { 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG "result": [ 10:14:10 DEBUG { 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "instance": "10.142.0.11:9100" 10:14:10 DEBUG }, 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG [...] 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "instance": "10.142.0.51:9100" 10:14:10 DEBUG }, 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG "67087.46666666667" 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG { 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 INFO [http_cmd]: http status code OK 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG --- status code --- 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 DEBUG "data": { 10:14:10 DEBUG { 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG --- http response --- 10:14:10 DEBUG { 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG "result": [ 10:14:10 DEBUG "data": { 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:14:10 DEBUG { 10:14:10 DEBUG { 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG "data": { 10:14:10 DEBUG ----- output ----- 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "result": [ 10:14:10 DEBUG { 10:14:10 DEBUG { 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG { 10:14:10 DEBUG "data": { 10:14:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:14:10 DEBUG { 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG "result": [ 10:14:10 DEBUG "status": "success", 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG }, 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "data": { 10:14:10 DEBUG "data": { 10:14:10 DEBUG { 10:14:10 DEBUG "data": { 10:14:10 DEBUG "result": [ 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "resultType": "vector", 10:14:10 DEBUG { 10:14:10 DEBUG 10:14:10 DEBUG }, 10:14:10 DEBUG "result": [ 10:14:10 DEBUG "result": [ 10:14:10 DEBUG "instance": "10.142.0.11:9100" 10:14:10 DEBUG "result": [ 10:14:10 DEBUG "metric": { 10:14:10 DEBUG 10:14:10 DEBUG "value": [ 10:14:10 DEBUG { 10:14:10 DEBUG { 10:14:10 DEBUG }, 10:14:10 DEBUG { 10:14:10 DEBUG "instance": "10.142.0.11:9100" 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "metric": { 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "metric": { 10:14:10 DEBUG }, 10:14:10 DEBUG [...] 10:14:10 DEBUG 10:14:10 DEBUG "instance": "10.142.0.11:9100" 10:14:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:14:10 DEBUG 10:14:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG }, 10:14:10 DEBUG }, 10:14:10 DEBUG 10:14:10 DEBUG }, 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG [...] 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG [...] 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "pod": "ds-idrepo-0" 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG [...] 10:14:10 DEBUG }, 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:14:10 DEBUG [...] 10:14:10 DEBUG [...] 10:14:10 DEBUG 10:14:10 DEBUG [...] 10:14:10 DEBUG 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG }, 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "instance": "10.142.0.51:9100" 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "0" 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG }, 10:14:10 DEBUG 10:14:10 DEBUG "instance": "10.142.0.51:9100" 10:14:10 DEBUG ] 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG 10:14:10 DEBUG 10:14:10 DEBUG "value": [ 10:14:10 DEBUG 10:14:10 DEBUG }, 10:14:10 DEBUG } 10:14:10 DEBUG "0.03957927741191452" 10:14:10 DEBUG "instance": "10.142.0.51:9100" 10:14:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG "pod": "ds-idrepo-0" 10:14:10 DEBUG "value": [ 10:14:10 DEBUG ] 10:14:10 DEBUG ] 10:14:10 DEBUG }, 10:14:10 DEBUG }, 10:14:10 DEBUG "0.00046666666666794754" 10:14:10 DEBUG }, 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "value": [ 10:14:10 DEBUG ] 10:14:10 DEBUG "value": [ 10:14:10 DEBUG "0.0009999999999990906" 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG } 10:14:10 DEBUG 1734776050, 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG "19797.333333333332" 10:14:10 DEBUG "15775948800" 10:14:10 DEBUG ] 10:14:10 DEBUG "0.0025264255901600876" 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG ] 10:14:10 DEBUG ] 10:14:10 DEBUG ] 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:10 DEBUG } 10:14:53 INFO 10:14:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:14:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:14:53 INFO [loop_until]: OK (rc = 0) 10:14:53 DEBUG --- stdout --- 10:14:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:14:53 DEBUG --- stderr --- 10:14:53 DEBUG 10:14:59 INFO 10:14:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:14:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:14:59 INFO [loop_until]: OK (rc = 0) 10:14:59 DEBUG --- stdout --- 10:14:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 107m 0% 3132Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 104m 0% 19858Mi 69% 10:14:59 DEBUG --- stderr --- 10:14:59 DEBUG 10:15:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776109" 10:15:09 INFO [http_cmd]: http status code OK 10:15:09 DEBUG --- status code --- 10:15:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:09 DEBUG --- http response --- 10:15:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776110" 10:15:10 ERROR [http_cmd]: ERROR 10:15:10 ERROR --- status code --- 10:15:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:15:10 ERROR --- http response --- 10:15:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:15:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG { 10:15:10 DEBUG { 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG "data": { 10:15:10 DEBUG "data": { 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "result": [ 10:15:10 DEBUG "result": [ 10:15:10 DEBUG { 10:15:10 DEBUG { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "pod": "ds-idrepo-0" 10:15:10 DEBUG "pod": "ds-cts-0" 10:15:10 DEBUG }, 10:15:10 DEBUG }, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG [...] 10:15:10 DEBUG [...] 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "pod": "ds-cts-0" 10:15:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:15:10 DEBUG }, 10:15:10 DEBUG }, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG "536.6700514265125" 10:15:10 DEBUG "732135424" 10:15:10 DEBUG ] 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG ] 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG { 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG "data": { 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "result": [ 10:15:10 DEBUG { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "instance": "10.142.0.11:9100" 10:15:10 DEBUG }, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG [...] 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "instance": "10.142.0.51:9100" 10:15:10 DEBUG }, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG "85062.23333333334" 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG { 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG "data": { 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "result": [ 10:15:10 DEBUG { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "instance": "10.142.0.11:9100" 10:15:10 DEBUG }, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG [...] 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "instance": "10.142.0.51:9100" 10:15:10 DEBUG }, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG "66780.50000000001" 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776110" 10:15:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776110" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:15:10 DEBUG ----- output ----- 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG { 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG { 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG { 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG "data": { 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG "data": { 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "data": { 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG "result": [ 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "result": [ 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG { 10:15:10 DEBUG "result": [ 10:15:10 DEBUG { 10:15:10 DEBUG { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:15:10 DEBUG "instance": "10.142.0.11:9100" 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG }, 10:15:10 DEBUG }, 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG "pod": "ds-idrepo-0" 10:15:10 DEBUG "data": { 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "value": [ 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG }, 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG "value": [ 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 INFO [http_cmd]: http status code OK 10:15:10 DEBUG "result": [ 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:15:10 DEBUG 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG --- status code --- 10:15:10 DEBUG { 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:15:10 DEBUG "metric": { 10:15:10 DEBUG [...] 10:15:10 DEBUG [...] 10:15:10 DEBUG { 10:15:10 DEBUG 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG --- http response --- 10:15:10 DEBUG "instance": "10.142.0.11:9100" 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG [...] 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:15:10 DEBUG }, 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "data": { 10:15:10 DEBUG 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG ----- output ----- 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG 10:15:10 DEBUG { 10:15:10 DEBUG { 10:15:10 DEBUG 10:15:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:15:10 DEBUG "instance": "10.142.0.51:9100" 10:15:10 DEBUG "result": [ 10:15:10 DEBUG 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG "status": "success", 10:15:10 DEBUG 10:15:10 DEBUG }, 10:15:10 DEBUG }, 10:15:10 DEBUG { 10:15:10 DEBUG "pod": "ds-cts-0" 10:15:10 DEBUG "data": { 10:15:10 DEBUG "data": { 10:15:10 DEBUG 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "metric": { 10:15:10 DEBUG }, 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG "resultType": "vector", 10:15:10 DEBUG [...] 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG "pod": "ds-idrepo-0" 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "result": [ 10:15:10 DEBUG "result": [ 10:15:10 DEBUG 10:15:10 DEBUG "0.044145422068546894" 10:15:10 DEBUG "0.000666666666666534" 10:15:10 DEBUG }, 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG { 10:15:10 DEBUG { 10:15:10 DEBUG 10:15:10 DEBUG ] 10:15:10 DEBUG ] 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "0" 10:15:10 DEBUG "metric": { 10:15:10 DEBUG "metric": { 10:15:10 DEBUG 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG 10:15:10 DEBUG ] 10:15:10 DEBUG "instance": "10.142.0.11:9100" 10:15:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:15:10 DEBUG "instance": "10.142.0.51:9100" 10:15:10 DEBUG ] 10:15:10 DEBUG ] 10:15:10 DEBUG 10:15:10 DEBUG } 10:15:10 DEBUG }, 10:15:10 DEBUG }, 10:15:10 DEBUG }, 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG 10:15:10 DEBUG ] 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "value": [ 10:15:10 DEBUG "value": [ 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG [...] 10:15:10 DEBUG } 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG 10:15:10 DEBUG } 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG "0.0005666666666655829" 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG ] 10:15:10 DEBUG 10:15:10 DEBUG [...] 10:15:10 DEBUG [...] 10:15:10 DEBUG } 10:15:10 DEBUG "pod": "ds-cts-0" 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG 10:15:10 DEBUG }, 10:15:10 DEBUG 10:15:10 DEBUG ] 10:15:10 DEBUG 10:15:10 DEBUG "value": [ 10:15:10 DEBUG 10:15:10 DEBUG } 10:15:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG "instance": "10.142.0.51:9100" 10:15:10 DEBUG } 10:15:10 DEBUG }, 10:15:10 DEBUG "0.0036792557895752934" 10:15:10 DEBUG ] 10:15:10 DEBUG "value": [ 10:15:10 DEBUG }, 10:15:10 DEBUG } 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG "value": [ 10:15:10 DEBUG ] 10:15:10 DEBUG "15775281152" 10:15:10 DEBUG 1734776110, 10:15:10 DEBUG } 10:15:10 DEBUG ] 10:15:10 DEBUG "21435.733333333334" 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG ] 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG ] 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:10 DEBUG } 10:15:53 INFO 10:15:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:15:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:15:53 INFO [loop_until]: OK (rc = 0) 10:15:53 DEBUG --- stdout --- 10:15:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:15:53 DEBUG --- stderr --- 10:15:53 DEBUG 10:15:59 INFO 10:15:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:15:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:15:59 INFO [loop_until]: OK (rc = 0) 10:15:59 DEBUG --- stdout --- 10:15:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 108m 0% 3141Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 103m 0% 19856Mi 69% 10:15:59 DEBUG --- stderr --- 10:15:59 DEBUG 10:16:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776169" 10:16:09 INFO [http_cmd]: http status code OK 10:16:09 DEBUG --- status code --- 10:16:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:09 DEBUG --- http response --- 10:16:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 ERROR [http_cmd]: ERROR 10:16:10 ERROR --- status code --- 10:16:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:16:10 ERROR --- http response --- 10:16:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:16:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG { 10:16:10 DEBUG { 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "data": { 10:16:10 DEBUG "data": { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG "result": [ 10:16:10 DEBUG { 10:16:10 DEBUG { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "pod": "ds-idrepo-0" 10:16:10 DEBUG "pod": "ds-cts-0" 10:16:10 DEBUG }, 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG [...] 10:16:10 DEBUG [...] 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:16:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:16:10 DEBUG }, 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG "559.1639026877685" 10:16:10 DEBUG "732135424" 10:16:10 DEBUG ] 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG ] 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG { 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "data": { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "instance": "10.142.0.11:9100" 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG [...] 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG "instance": "10.142.0.51:9100" 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG "87607.00000000003" 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG { 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "data": { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "instance": "10.142.0.11:9100" 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG [...] 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG "instance": "10.142.0.51:9100" 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG "72053.6" 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776170" 10:16:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776170" 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG { 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "data": { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "instance": "10.142.0.11:9100" 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG [...] 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG --- http response --- 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 INFO [http_cmd]: http status code OK 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG --- status code --- 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:16:10 DEBUG 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG 10:16:10 DEBUG { 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG --- http response --- 10:16:10 DEBUG { 10:16:10 DEBUG "instance": "10.142.0.51:9100" 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG }, 10:16:10 DEBUG "data": { 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG "data": { 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG ----- output ----- 10:16:10 DEBUG "data": { 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG { 10:16:10 DEBUG { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG "result": [ 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "status": "success", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "pod": "ds-idrepo-0" 10:16:10 DEBUG }, 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "data": { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "result": [ 10:16:10 DEBUG "data": { 10:16:10 DEBUG { 10:16:10 DEBUG "data": { 10:16:10 DEBUG 10:16:10 DEBUG "0.0003333333333330302" 10:16:10 DEBUG { 10:16:10 DEBUG { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "resultType": "vector", 10:16:10 DEBUG 10:16:10 DEBUG ] 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "result": [ 10:16:10 DEBUG "pod": "ds-idrepo-0" 10:16:10 DEBUG "result": [ 10:16:10 DEBUG 10:16:10 DEBUG } 10:16:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:16:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:16:10 DEBUG { 10:16:10 DEBUG }, 10:16:10 DEBUG { 10:16:10 DEBUG [...] 10:16:10 DEBUG ] 10:16:10 DEBUG }, 10:16:10 DEBUG }, 10:16:10 DEBUG "metric": { 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "metric": { 10:16:10 DEBUG 10:16:10 DEBUG } 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "instance": "10.142.0.11:9100" 10:16:10 DEBUG 10:16:10 DEBUG "instance": "10.142.0.11:9100" 10:16:10 DEBUG 10:16:10 DEBUG } 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG }, 10:16:10 DEBUG 10:16:10 DEBUG }, 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG [...] 10:16:10 DEBUG 10:16:10 DEBUG }, 10:16:10 DEBUG [...] 10:16:10 DEBUG [...] 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG [...] 10:16:10 DEBUG 10:16:10 DEBUG [...] 10:16:10 DEBUG "0.0011430869739263861" 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG 10:16:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:16:10 DEBUG 10:16:10 DEBUG ] 10:16:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:16:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:16:10 DEBUG 10:16:10 DEBUG }, 10:16:10 DEBUG 10:16:10 DEBUG } 10:16:10 DEBUG }, 10:16:10 DEBUG }, 10:16:10 DEBUG 10:16:10 DEBUG "value": [ 10:16:10 DEBUG 10:16:10 DEBUG ] 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "value": [ 10:16:10 DEBUG "instance": "10.142.0.51:9100" 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG "instance": "10.142.0.51:9100" 10:16:10 DEBUG } 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG }, 10:16:10 DEBUG "0" 10:16:10 DEBUG }, 10:16:10 DEBUG } 10:16:10 DEBUG "0.04032640353742322" 10:16:10 DEBUG "15772753920" 10:16:10 DEBUG "value": [ 10:16:10 DEBUG ] 10:16:10 DEBUG "value": [ 10:16:10 DEBUG ] 10:16:10 DEBUG ] 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG } 10:16:10 DEBUG 1734776170, 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG "0.00040000000000001514" 10:16:10 DEBUG ] 10:16:10 DEBUG "23620.266666666666" 10:16:10 DEBUG ] 10:16:10 DEBUG ] 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG ] 10:16:10 DEBUG ] 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:10 DEBUG } 10:16:53 INFO 10:16:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:16:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:16:53 INFO [loop_until]: OK (rc = 0) 10:16:53 DEBUG --- stdout --- 10:16:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:16:53 DEBUG --- stderr --- 10:16:53 DEBUG 10:16:59 INFO 10:16:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:16:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:16:59 INFO [loop_until]: OK (rc = 0) 10:16:59 DEBUG --- stdout --- 10:16:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 105m 0% 3163Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19858Mi 69% 10:16:59 DEBUG --- stderr --- 10:16:59 DEBUG 10:17:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776229" 10:17:09 INFO [http_cmd]: http status code OK 10:17:09 DEBUG --- status code --- 10:17:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:09 DEBUG --- http response --- 10:17:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG --- http response --- 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 ERROR [http_cmd]: ERROR 10:17:10 ERROR --- status code --- 10:17:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:17:10 ERROR --- http response --- 10:17:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:17:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG { 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG "data": { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "result": [ 10:17:10 DEBUG { 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "pod": "ds-cts-0" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG "1270.6492083995597" 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG { 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG "data": { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "result": [ 10:17:10 DEBUG { 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "pod": "ds-cts-0" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG "732315648" 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG { 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG "data": { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "result": [ 10:17:10 DEBUG { 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "instance": "10.142.0.11:9100" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "instance": "10.142.0.51:9100" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG "86016.16666666666" 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG { 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG "data": { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "result": [ 10:17:10 DEBUG { 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "instance": "10.142.0.11:9100" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "instance": "10.142.0.51:9100" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG "67730.13333333335" 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776230" 10:17:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776230" 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG --- http response --- 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG { 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG "data": { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG "result": [ 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG { 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG { 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG "data": { 10:17:10 DEBUG "metric": { 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "instance": "10.142.0.11:9100" 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 INFO [http_cmd]: http status code OK 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG "result": [ 10:17:10 DEBUG }, 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG --- status code --- 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG { 10:17:10 DEBUG "value": [ 10:17:10 DEBUG { 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:17:10 DEBUG { 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG "metric": { 10:17:10 DEBUG 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG --- http response --- 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG "pod": "ds-cts-0" 10:17:10 DEBUG 10:17:10 DEBUG "data": { 10:17:10 DEBUG { 10:17:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:17:10 DEBUG "data": { 10:17:10 DEBUG { 10:17:10 DEBUG }, 10:17:10 DEBUG 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG ----- output ----- 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG "value": [ 10:17:10 DEBUG [...] 10:17:10 DEBUG "result": [ 10:17:10 DEBUG "data": { 10:17:10 DEBUG { 10:17:10 DEBUG "result": [ 10:17:10 DEBUG "data": { 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "status": "success", 10:17:10 DEBUG { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "pod": "ds-cts-0" 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "result": [ 10:17:10 DEBUG "data": { 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "result": [ 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG "instance": "10.142.0.51:9100" 10:17:10 DEBUG { 10:17:10 DEBUG "resultType": "vector", 10:17:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:17:10 DEBUG { 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG }, 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "result": [ 10:17:10 DEBUG }, 10:17:10 DEBUG "metric": { 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG "value": [ 10:17:10 DEBUG "instance": "10.142.0.11:9100" 10:17:10 DEBUG { 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG }, 10:17:10 DEBUG "metric": { 10:17:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG 10:17:10 DEBUG "value": [ 10:17:10 DEBUG "instance": "10.142.0.11:9100" 10:17:10 DEBUG }, 10:17:10 DEBUG 10:17:10 DEBUG "pod": "ds-idrepo-0" 10:17:10 DEBUG "0.0009999999999995642" 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG }, 10:17:10 DEBUG ] 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:17:10 DEBUG "value": [ 10:17:10 DEBUG } 10:17:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG }, 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG ] 10:17:10 DEBUG }, 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG "value": [ 10:17:10 DEBUG "0" 10:17:10 DEBUG } 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG 10:17:10 DEBUG [...] 10:17:10 DEBUG 10:17:10 DEBUG "0" 10:17:10 DEBUG } 10:17:10 DEBUG "0.044554439598990206" 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG ] 10:17:10 DEBUG ] 10:17:10 DEBUG ] 10:17:10 DEBUG "instance": "10.142.0.51:9100" 10:17:10 DEBUG 10:17:10 DEBUG 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG }, 10:17:10 DEBUG 10:17:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG "value": [ 10:17:10 DEBUG "instance": "10.142.0.51:9100" 10:17:10 DEBUG }, 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG }, 10:17:10 DEBUG "value": [ 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG "0.0012666666666641883" 10:17:10 DEBUG "value": [ 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG ] 10:17:10 DEBUG 1734776230, 10:17:10 DEBUG "15774633984" 10:17:10 DEBUG } 10:17:10 DEBUG "37137.066666666666" 10:17:10 DEBUG ] 10:17:10 DEBUG ] 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG ] 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:10 DEBUG } 10:17:53 INFO 10:17:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:17:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:17:53 INFO [loop_until]: OK (rc = 0) 10:17:53 DEBUG --- stdout --- 10:17:53 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 5m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:17:53 DEBUG --- stderr --- 10:17:53 DEBUG 10:17:59 INFO 10:17:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:17:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:17:59 INFO [loop_until]: OK (rc = 0) 10:17:59 DEBUG --- stdout --- 10:17:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 110m 0% 3181Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 101m 0% 19860Mi 69% 10:17:59 DEBUG --- stderr --- 10:17:59 DEBUG 10:18:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776289" 10:18:09 INFO [http_cmd]: http status code OK 10:18:09 DEBUG --- status code --- 10:18:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:09 DEBUG --- http response --- 10:18:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 ERROR [http_cmd]: ERROR 10:18:10 ERROR --- status code --- 10:18:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:18:10 ERROR --- http response --- 10:18:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:18:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG { 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG "data": { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "result": [ 10:18:10 DEBUG { 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "pod": "ds-cts-0" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG "443.7382637584862" 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG { 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG "data": { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "result": [ 10:18:10 DEBUG { 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "pod": "ds-cts-0" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG "732127232" 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG { 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG "data": { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "result": [ 10:18:10 DEBUG { 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "instance": "10.142.0.11:9100" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "instance": "10.142.0.51:9100" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG "88374.79999999999" 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG { 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG "data": { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "result": [ 10:18:10 DEBUG { 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "instance": "10.142.0.11:9100" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "instance": "10.142.0.51:9100" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG "70025.3666666667" 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776290" 10:18:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776290" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG { 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG "data": { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "result": [ 10:18:10 DEBUG { 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "instance": "10.142.0.11:9100" 10:18:10 DEBUG }, 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "instance": "10.142.0.51:9100" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG }, 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG "value": [ 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG "0.00033333333333350387" 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:18:10 DEBUG ] 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG } 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG { 10:18:10 DEBUG { 10:18:10 DEBUG ] 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG "status": "success", 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG } 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG "data": { 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG "data": { 10:18:10 DEBUG } 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG "result": [ 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG "result": [ 10:18:10 DEBUG { 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG { 10:18:10 DEBUG { 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "metric": { 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG "data": { 10:18:10 DEBUG { 10:18:10 DEBUG "pod": "ds-cts-0" 10:18:10 INFO [http_cmd]: http status code OK 10:18:10 DEBUG "pod": "ds-cts-0" 10:18:10 DEBUG { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG }, 10:18:10 DEBUG --- status code --- 10:18:10 DEBUG }, 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG "result": [ 10:18:10 DEBUG "data": { 10:18:10 DEBUG "value": [ 10:18:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:18:10 DEBUG "value": [ 10:18:10 DEBUG "data": { 10:18:10 DEBUG { 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG 10:18:10 DEBUG --- http response --- 10:18:10 DEBUG 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "result": [ 10:18:10 DEBUG 10:18:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:18:10 DEBUG 10:18:10 DEBUG "result": [ 10:18:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:18:10 DEBUG { 10:18:10 DEBUG 10:18:10 DEBUG ----- output ----- 10:18:10 DEBUG 10:18:10 DEBUG { 10:18:10 DEBUG }, 10:18:10 DEBUG "metric": { 10:18:10 DEBUG [...] 10:18:10 DEBUG { 10:18:10 DEBUG [...] 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "value": [ 10:18:10 DEBUG "instance": "10.142.0.11:9100" 10:18:10 DEBUG 10:18:10 DEBUG "status": "success", 10:18:10 DEBUG 10:18:10 DEBUG "instance": "10.142.0.11:9100" 10:18:10 DEBUG 10:18:10 DEBUG }, 10:18:10 DEBUG 10:18:10 DEBUG "data": { 10:18:10 DEBUG 10:18:10 DEBUG }, 10:18:10 DEBUG 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG "resultType": "vector", 10:18:10 DEBUG 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:18:10 DEBUG "result": [ 10:18:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG }, 10:18:10 DEBUG { 10:18:10 DEBUG }, 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "value": [ 10:18:10 DEBUG "metric": { 10:18:10 DEBUG "value": [ 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG [...] 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG [...] 10:18:10 DEBUG 10:18:10 DEBUG 10:18:10 DEBUG "0" 10:18:10 DEBUG }, 10:18:10 DEBUG "0.0006088762097379254" 10:18:10 DEBUG 10:18:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:18:10 DEBUG 10:18:10 DEBUG ] 10:18:10 DEBUG "value": [ 10:18:10 DEBUG ] 10:18:10 DEBUG 10:18:10 DEBUG }, 10:18:10 DEBUG 10:18:10 DEBUG } 10:18:10 DEBUG 10:18:10 DEBUG } 10:18:10 DEBUG 10:18:10 DEBUG "value": [ 10:18:10 DEBUG "instance": "10.142.0.51:9100" 10:18:10 DEBUG ] 10:18:10 DEBUG 10:18:10 DEBUG ] 10:18:10 DEBUG "instance": "10.142.0.51:9100" 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG }, 10:18:10 DEBUG } 10:18:10 DEBUG 10:18:10 DEBUG } 10:18:10 DEBUG }, 10:18:10 DEBUG "0.046767802337863276" 10:18:10 DEBUG "value": [ 10:18:10 DEBUG } 10:18:10 DEBUG [...] 10:18:10 DEBUG } 10:18:10 DEBUG "value": [ 10:18:10 DEBUG ] 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG } 10:18:10 DEBUG "31675.73333333333" 10:18:10 DEBUG 10:18:10 DEBUG "0.0005333333333311429" 10:18:10 DEBUG ] 10:18:10 DEBUG ] 10:18:10 DEBUG 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 DEBUG ] 10:18:10 DEBUG }, 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG "value": [ 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:10 DEBUG 1734776290, 10:18:10 DEBUG } 10:18:10 DEBUG "15776411648" 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG ] 10:18:10 DEBUG } 10:18:10 DEBUG } 10:18:53 INFO 10:18:53 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:18:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:18:54 INFO [loop_until]: OK (rc = 0) 10:18:54 DEBUG --- stdout --- 10:18:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:18:54 DEBUG --- stderr --- 10:18:54 DEBUG 10:18:59 INFO 10:18:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:18:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:18:59 INFO [loop_until]: OK (rc = 0) 10:18:59 DEBUG --- stdout --- 10:18:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 107m 0% 3172Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 100m 0% 19857Mi 69% 10:18:59 DEBUG --- stderr --- 10:18:59 DEBUG 10:19:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776349" 10:19:09 INFO [http_cmd]: http status code OK 10:19:09 DEBUG --- status code --- 10:19:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:09 DEBUG --- http response --- 10:19:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG --- status code --- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 ERROR [http_cmd]: ERROR 10:19:10 ERROR --- status code --- 10:19:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:19:10 ERROR --- http response --- 10:19:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:19:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG --- http response --- 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "data": { 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "result": [ 10:19:10 DEBUG { 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "pod": "ds-cts-0" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG [...] 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG "416.1051098845282" 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "data": { 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "result": [ 10:19:10 DEBUG { 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "pod": "ds-cts-0" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG [...] 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG "732127232" 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "data": { 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "result": [ 10:19:10 DEBUG { 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "instance": "10.142.0.11:9100" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG [...] 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "instance": "10.142.0.51:9100" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG "85653.53333333334" 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "data": { 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "result": [ 10:19:10 DEBUG { 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "instance": "10.142.0.11:9100" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG [...] 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "instance": "10.142.0.51:9100" 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG "67563.83333333333" 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776350" 10:19:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776350" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "data": { 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "result": [ 10:19:10 DEBUG { 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "instance": "10.142.0.11:9100" 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG }, 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG "value": [ 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:19:10 DEBUG 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG [...] 10:19:10 DEBUG { 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG { 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG 10:19:10 DEBUG "data": { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "data": { 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG "instance": "10.142.0.51:9100" 10:19:10 DEBUG "result": [ 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG ----- output ----- 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 INFO [http_cmd]: http status code OK 10:19:10 DEBUG }, 10:19:10 DEBUG { 10:19:10 DEBUG "result": [ 10:19:10 DEBUG { 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG --- status code --- 10:19:10 DEBUG "value": [ 10:19:10 DEBUG "metric": { 10:19:10 DEBUG { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG "pod": "ds-cts-0" 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "data": { 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG --- http response --- 10:19:10 DEBUG "0" 10:19:10 DEBUG }, 10:19:10 DEBUG "pod": "ds-cts-0" 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:19:10 DEBUG ] 10:19:10 DEBUG "value": [ 10:19:10 DEBUG }, 10:19:10 DEBUG "result": [ 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG ----- output ----- 10:19:10 DEBUG } 10:19:10 DEBUG 10:19:10 DEBUG "value": [ 10:19:10 DEBUG { 10:19:10 DEBUG { 10:19:10 DEBUG { 10:19:10 DEBUG { 10:19:10 DEBUG ] 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG "status": "success", 10:19:10 DEBUG } 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:19:10 DEBUG "data": { 10:19:10 DEBUG "data": { 10:19:10 DEBUG "data": { 10:19:10 DEBUG } 10:19:10 DEBUG [...] 10:19:10 DEBUG 10:19:10 DEBUG }, 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG "resultType": "vector", 10:19:10 DEBUG 10:19:10 DEBUG [...] 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 10:19:10 DEBUG "result": [ 10:19:10 DEBUG "result": [ 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG "result": [ 10:19:10 DEBUG 10:19:10 DEBUG { 10:19:10 DEBUG { 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG { 10:19:10 DEBUG 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "metric": { 10:19:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:19:10 DEBUG 10:19:10 DEBUG "metric": { 10:19:10 DEBUG [...] 10:19:10 DEBUG "instance": "10.142.0.11:9100" 10:19:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:19:10 DEBUG }, 10:19:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:19:10 DEBUG "instance": "10.142.0.11:9100" 10:19:10 DEBUG 10:19:10 DEBUG }, 10:19:10 DEBUG }, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG }, 10:19:10 DEBUG }, 10:19:10 DEBUG 10:19:10 DEBUG "value": [ 10:19:10 DEBUG "value": [ 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG "value": [ 10:19:10 DEBUG "value": [ 10:19:10 DEBUG [...] 10:19:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:19:10 DEBUG 10:19:10 DEBUG "0.0008758018385569344" 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG }, 10:19:10 DEBUG 10:19:10 DEBUG ] 10:19:10 DEBUG "0" 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG [...] 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG "value": [ 10:19:10 DEBUG [...] 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG "0.037476753206719414" 10:19:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:19:10 DEBUG }, 10:19:10 DEBUG 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG 10:19:10 DEBUG "value": [ 10:19:10 DEBUG "instance": "10.142.0.51:9100" 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG "instance": "10.142.0.51:9100" 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG }, 10:19:10 DEBUG ] 10:19:10 DEBUG }, 10:19:10 DEBUG "15777075200" 10:19:10 DEBUG "value": [ 10:19:10 DEBUG } 10:19:10 DEBUG "value": [ 10:19:10 DEBUG ] 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG } 10:19:10 DEBUG 1734776350, 10:19:10 DEBUG } 10:19:10 DEBUG "19660.8" 10:19:10 DEBUG "0.00039999999999906777" 10:19:10 DEBUG ] 10:19:10 DEBUG ] 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG ] 10:19:10 DEBUG ] 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:10 DEBUG } 10:19:54 INFO 10:19:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:19:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:19:54 INFO [loop_until]: OK (rc = 0) 10:19:54 DEBUG --- stdout --- 10:19:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:19:54 DEBUG --- stderr --- 10:19:54 DEBUG 10:19:59 INFO 10:19:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:19:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:19:59 INFO [loop_until]: OK (rc = 0) 10:19:59 DEBUG --- stdout --- 10:19:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 106m 0% 3166Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 102m 0% 19859Mi 69% 10:19:59 DEBUG --- stderr --- 10:19:59 DEBUG 10:20:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776409" 10:20:09 INFO [http_cmd]: http status code OK 10:20:09 DEBUG --- status code --- 10:20:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:09 DEBUG --- http response --- 10:20:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 ERROR [http_cmd]: ERROR 10:20:10 ERROR --- status code --- 10:20:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:20:10 ERROR --- http response --- 10:20:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:20:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG { 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG "data": { 10:20:10 DEBUG { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG "data": { 10:20:10 DEBUG { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "result": [ 10:20:10 DEBUG "pod": "ds-cts-0" 10:20:10 DEBUG { 10:20:10 DEBUG }, 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "value": [ 10:20:10 DEBUG "pod": "ds-cts-0" 10:20:10 DEBUG 10:20:10 DEBUG }, 10:20:10 DEBUG 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:20:10 DEBUG 10:20:10 DEBUG }, 10:20:10 DEBUG 10:20:10 DEBUG "value": [ 10:20:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG }, 10:20:10 DEBUG "444.69777162554624" 10:20:10 DEBUG "value": [ 10:20:10 DEBUG ] 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG } 10:20:10 DEBUG "732127232" 10:20:10 DEBUG ] 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG { 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG { 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "instance": "10.142.0.11:9100" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG "instance": "10.142.0.51:9100" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG "85254.56666666668" 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG { 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG { 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "instance": "10.142.0.11:9100" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG "instance": "10.142.0.51:9100" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG "66996.26666666668" 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776410" 10:20:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776410" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG { 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG { 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "instance": "10.142.0.11:9100" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG "instance": "10.142.0.51:9100" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG "0.00033333333333350387" 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG { 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG { 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "pod": "ds-cts-0" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG "0" 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG { 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG { 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:20:10 DEBUG }, 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG [...] 10:20:10 DEBUG 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:20:10 DEBUG 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG 10:20:10 DEBUG { 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:20:10 INFO [http_cmd]: http status code OK 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG --- status code --- 10:20:10 DEBUG }, 10:20:10 DEBUG { 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG "value": [ 10:20:10 DEBUG "metric": { 10:20:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG "pod": "ds-cts-0" 10:20:10 DEBUG --- http response --- 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG "15772467200" 10:20:10 DEBUG }, 10:20:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:20:10 DEBUG { 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG ] 10:20:10 DEBUG "value": [ 10:20:10 DEBUG ----- output ----- 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG { 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG "data": { 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG "result": [ 10:20:10 DEBUG { 10:20:10 DEBUG "metric": { 10:20:10 DEBUG "data": { 10:20:10 DEBUG } 10:20:10 DEBUG 10:20:10 DEBUG { 10:20:10 DEBUG "instance": "10.142.0.11:9100" 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG ] 10:20:10 DEBUG 10:20:10 DEBUG "status": "success", 10:20:10 DEBUG }, 10:20:10 DEBUG "result": [ 10:20:10 DEBUG } 10:20:10 DEBUG 10:20:10 DEBUG "data": { 10:20:10 DEBUG "value": [ 10:20:10 DEBUG { 10:20:10 DEBUG } 10:20:10 DEBUG [...] 10:20:10 DEBUG "resultType": "vector", 10:20:10 DEBUG 10:20:10 DEBUG "metric": { 10:20:10 DEBUG 10:20:10 DEBUG "result": [ 10:20:10 DEBUG 10:20:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:20:10 DEBUG 10:20:10 DEBUG { 10:20:10 DEBUG 10:20:10 DEBUG }, 10:20:10 DEBUG 10:20:10 DEBUG "metric": { 10:20:10 DEBUG [...] 10:20:10 DEBUG "value": [ 10:20:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:20:10 DEBUG "instance": "10.142.0.11:9100" 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG }, 10:20:10 DEBUG }, 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG "value": [ 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG 10:20:10 DEBUG "instance": "10.142.0.51:9100" 10:20:10 DEBUG [...] 10:20:10 DEBUG "0.0009619231689965589" 10:20:10 DEBUG 10:20:10 DEBUG }, 10:20:10 DEBUG 10:20:10 DEBUG ] 10:20:10 DEBUG 10:20:10 DEBUG "value": [ 10:20:10 DEBUG 10:20:10 DEBUG } 10:20:10 DEBUG [...] 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG 10:20:10 DEBUG ] 10:20:10 DEBUG 10:20:10 DEBUG "22391.466666666667" 10:20:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:20:10 DEBUG } 10:20:10 DEBUG 10:20:10 DEBUG ] 10:20:10 DEBUG }, 10:20:10 DEBUG } 10:20:10 DEBUG 10:20:10 DEBUG } 10:20:10 DEBUG "value": [ 10:20:10 DEBUG "instance": "10.142.0.51:9100" 10:20:10 DEBUG ] 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG }, 10:20:10 DEBUG } 10:20:10 DEBUG "0.042375465550010996" 10:20:10 DEBUG "value": [ 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG 1734776410, 10:20:10 DEBUG } 10:20:10 DEBUG "0.0004333333333325603" 10:20:10 DEBUG ] 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:10 DEBUG ] 10:20:10 DEBUG } 10:20:10 DEBUG } 10:20:54 INFO 10:20:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:20:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:20:54 INFO [loop_until]: OK (rc = 0) 10:20:54 DEBUG --- stdout --- 10:20:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:20:54 DEBUG --- stderr --- 10:20:54 DEBUG 10:20:59 INFO 10:20:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:20:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:20:59 INFO [loop_until]: OK (rc = 0) 10:20:59 DEBUG --- stdout --- 10:20:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 103m 0% 3172Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 102m 0% 19861Mi 69% 10:20:59 DEBUG --- stderr --- 10:20:59 DEBUG 10:21:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776469" 10:21:09 INFO [http_cmd]: http status code OK 10:21:09 DEBUG --- status code --- 10:21:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:09 DEBUG --- http response --- 10:21:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776470" 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 ERROR [http_cmd]: ERROR 10:21:10 ERROR --- status code --- 10:21:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:21:10 ERROR --- http response --- 10:21:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:21:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "pod": "ds-cts-0" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "732127232" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "pod": "ds-cts-0" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "106.54226027902855" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "instance": "10.142.0.11:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "instance": "10.142.0.51:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "87143.43333333333" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "instance": "10.142.0.11:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "instance": "10.142.0.51:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "75254.00000000003" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "instance": "10.142.0.11:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "instance": "10.142.0.51:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "0.00033333333333350387" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "pod": "ds-cts-0" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "0" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776470" 10:21:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776470" 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG { 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "data": { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "15773741056" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- http response --- 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG { 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG "data": { 10:21:10 DEBUG { 10:21:10 DEBUG { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "result": [ 10:21:10 DEBUG "data": { 10:21:10 DEBUG "data": { 10:21:10 DEBUG { 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "result": [ 10:21:10 DEBUG "result": [ 10:21:10 DEBUG "instance": "10.142.0.11:9100" 10:21:10 DEBUG { 10:21:10 DEBUG { 10:21:10 DEBUG }, 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "metric": { 10:21:10 DEBUG "value": [ 10:21:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:21:10 DEBUG "pod": "ds-cts-0" 10:21:10 DEBUG 10:21:10 DEBUG }, 10:21:10 DEBUG }, 10:21:10 DEBUG 10:21:10 DEBUG "value": [ 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 INFO [http_cmd]: http status code OK 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG --- status code --- 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG [...] 10:21:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG --- http response --- 10:21:10 DEBUG "instance": "10.142.0.51:9100" 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:21:10 DEBUG }, 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG ----- output ----- 10:21:10 DEBUG "value": [ 10:21:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:21:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:21:10 DEBUG { 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG }, 10:21:10 DEBUG }, 10:21:10 DEBUG "status": "success", 10:21:10 DEBUG "26897.06666666667" 10:21:10 DEBUG "value": [ 10:21:10 DEBUG "value": [ 10:21:10 DEBUG "data": { 10:21:10 DEBUG ] 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "resultType": "vector", 10:21:10 DEBUG } 10:21:10 DEBUG "0.0386408555747651" 10:21:10 DEBUG "0.00043574119731297413" 10:21:10 DEBUG "result": [ 10:21:10 DEBUG ] 10:21:10 DEBUG ] 10:21:10 DEBUG ] 10:21:10 DEBUG { 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 DEBUG "metric": { 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG ] 10:21:10 DEBUG "instance": "10.142.0.11:9100" 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 DEBUG }, 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG [...] 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG 10:21:10 DEBUG "instance": "10.142.0.51:9100" 10:21:10 DEBUG }, 10:21:10 DEBUG "value": [ 10:21:10 DEBUG 1734776470, 10:21:10 DEBUG "0.0004999999999976504" 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG ] 10:21:10 DEBUG } 10:21:10 DEBUG } 10:21:54 INFO 10:21:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:21:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:21:54 INFO [loop_until]: OK (rc = 0) 10:21:54 DEBUG --- stdout --- 10:21:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:21:54 DEBUG --- stderr --- 10:21:54 DEBUG 10:21:59 INFO 10:21:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:21:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:21:59 INFO [loop_until]: OK (rc = 0) 10:21:59 DEBUG --- stdout --- 10:21:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 99m 0% 3157Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 104m 0% 19866Mi 69% 10:21:59 DEBUG --- stderr --- 10:21:59 DEBUG 10:22:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776529" 10:22:09 INFO [http_cmd]: http status code OK 10:22:09 DEBUG --- status code --- 10:22:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:09 DEBUG --- http response --- 10:22:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 ERROR [http_cmd]: ERROR 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 ERROR --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 ERROR --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:22:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "pod": "ds-cts-0" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "732127232" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "pod": "ds-idrepo-0" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "103.1051540536703" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "instance": "10.142.0.11:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "instance": "10.142.0.51:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "87451.93333333333" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "instance": "10.142.0.11:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "instance": "10.142.0.51:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "72163.30000000002" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "instance": "10.142.0.11:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "instance": "10.142.0.51:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "0.001000000000000038" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776530" 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "pod": "ds-idrepo-0" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "0" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776530" 10:22:10 DEBUG 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776530" 10:22:10 DEBUG 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776530" 10:22:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776530" 10:22:10 DEBUG 10:22:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "15779037184" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "data": { 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "instance": "10.142.0.11:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG "instance": "10.142.0.51:9100" 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG }, 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG "value": [ 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG "0.0011666666666684477" 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG ] 10:22:10 DEBUG { 10:22:10 DEBUG } 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG ] 10:22:10 DEBUG "data": { 10:22:10 DEBUG } 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG } 10:22:10 DEBUG "result": [ 10:22:10 DEBUG { 10:22:10 DEBUG "metric": { 10:22:10 DEBUG "instance": "10.142.0.11:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG [...] 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG 10:22:10 DEBUG "instance": "10.142.0.51:9100" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "25804.8" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG { 10:22:10 DEBUG "status": "success", 10:22:10 INFO [http_cmd]: http status code OK 10:22:10 DEBUG "data": { 10:22:10 DEBUG --- status code --- 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:22:10 DEBUG "result": [ 10:22:10 DEBUG --- http response --- 10:22:10 DEBUG { 10:22:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:22:10 DEBUG "metric": { 10:22:10 DEBUG ----- output ----- 10:22:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:22:10 DEBUG { 10:22:10 DEBUG }, 10:22:10 DEBUG "status": "success", 10:22:10 DEBUG "value": [ 10:22:10 DEBUG "data": { 10:22:10 DEBUG 10:22:10 DEBUG "resultType": "vector", 10:22:10 DEBUG 10:22:10 DEBUG "result": [ 10:22:10 DEBUG 10:22:10 DEBUG { 10:22:10 DEBUG [...] 10:22:10 DEBUG "metric": { 10:22:10 DEBUG 10:22:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:22:10 DEBUG 10:22:10 DEBUG }, 10:22:10 DEBUG 10:22:10 DEBUG "value": [ 10:22:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:22:10 DEBUG 10:22:10 DEBUG }, 10:22:10 DEBUG 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG [...] 10:22:10 DEBUG "0.0009401800546657257" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG 10:22:10 DEBUG ] 10:22:10 DEBUG 10:22:10 DEBUG } 10:22:10 DEBUG 10:22:10 DEBUG } 10:22:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:22:10 DEBUG }, 10:22:10 DEBUG "value": [ 10:22:10 DEBUG 1734776530, 10:22:10 DEBUG "0.03529752934298898" 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG ] 10:22:10 DEBUG } 10:22:10 DEBUG } 10:22:54 INFO 10:22:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:22:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:22:54 INFO [loop_until]: OK (rc = 0) 10:22:54 DEBUG --- stdout --- 10:22:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:22:54 DEBUG --- stderr --- 10:22:54 DEBUG 10:22:59 INFO 10:22:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:22:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:22:59 INFO [loop_until]: OK (rc = 0) 10:22:59 DEBUG --- stdout --- 10:22:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 98m 0% 3168Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19857Mi 69% 10:22:59 DEBUG --- stderr --- 10:22:59 DEBUG 10:23:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776589" 10:23:09 INFO [http_cmd]: http status code OK 10:23:09 DEBUG --- status code --- 10:23:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:09 DEBUG --- http response --- 10:23:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 ERROR [http_cmd]: ERROR 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 ERROR --- status code --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 ERROR --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "pod": "ds-idrepo-0" 10:23:10 DEBUG "pod": "ds-cts-0" 10:23:10 DEBUG }, 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:23:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:23:10 DEBUG }, 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "429.9282433581999" 10:23:10 DEBUG "732127232" 10:23:10 DEBUG ] 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "instance": "10.142.0.11:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "instance": "10.142.0.51:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "86750.26666666668" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "instance": "10.142.0.11:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "instance": "10.142.0.51:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "71255.53333333334" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "instance": "10.142.0.11:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "instance": "10.142.0.51:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "0" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "pod": "ds-idrepo-0" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "0" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "15771033600" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776590" 10:23:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776590" 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "instance": "10.142.0.11:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "instance": "10.142.0.51:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "0.0004666666666688949" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "instance": "10.142.0.11:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "instance": "10.142.0.51:9100" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "22391.466666666667" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "pod": "ds-idrepo-0" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "pod": "ds-cts-0" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "0" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:10 INFO [http_cmd]: http status code OK 10:23:10 DEBUG --- status code --- 10:23:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:23:10 DEBUG --- http response --- 10:23:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:23:10 DEBUG ----- output ----- 10:23:10 DEBUG { 10:23:10 DEBUG "status": "success", 10:23:10 DEBUG "data": { 10:23:10 DEBUG "resultType": "vector", 10:23:10 DEBUG "result": [ 10:23:10 DEBUG { 10:23:10 DEBUG "metric": { 10:23:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG [...] 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG 10:23:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:23:10 DEBUG }, 10:23:10 DEBUG "value": [ 10:23:10 DEBUG 1734776590, 10:23:10 DEBUG "0.04104820889375095" 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG ] 10:23:10 DEBUG } 10:23:10 DEBUG } 10:23:54 INFO 10:23:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:23:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:23:54 INFO [loop_until]: OK (rc = 0) 10:23:54 DEBUG --- stdout --- 10:23:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:23:54 DEBUG --- stderr --- 10:23:54 DEBUG 10:23:59 INFO 10:23:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:23:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:23:59 INFO [loop_until]: OK (rc = 0) 10:23:59 DEBUG --- stdout --- 10:23:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 106m 0% 3173Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 94m 0% 19854Mi 69% 10:23:59 DEBUG --- stderr --- 10:23:59 DEBUG 10:24:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776649" 10:24:09 INFO [http_cmd]: http status code OK 10:24:09 DEBUG --- status code --- 10:24:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:09 DEBUG --- http response --- 10:24:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- status code --- 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 ERROR [http_cmd]: ERROR 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 DEBUG --- http response --- 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 ERROR --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- http response --- 10:24:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 ERROR --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "pod": "ds-cts-0" 10:24:10 DEBUG "pod": "ds-cts-0" 10:24:10 DEBUG }, 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:24:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:24:10 DEBUG }, 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "559.598087057543" 10:24:10 DEBUG "732127232" 10:24:10 DEBUG ] 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "instance": "10.142.0.11:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "instance": "10.142.0.51:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "85387.99999999999" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "instance": "10.142.0.11:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "instance": "10.142.0.51:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "68992.53333333334" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "instance": "10.142.0.11:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "instance": "10.142.0.51:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "0.0013333333333335419" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "pod": "ds-cts-0" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "0" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776650" 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776650" 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776650" 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "15772393472" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776650" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "instance": "10.142.0.11:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "instance": "10.142.0.51:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "0.0012000000000000454" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "instance": "10.142.0.11:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "instance": "10.142.0.51:9100" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "24166.4" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "pod": "ds-cts-0" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "pod": "ds-idrepo-0" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "0" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:10 INFO [http_cmd]: http status code OK 10:24:10 DEBUG --- status code --- 10:24:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:24:10 DEBUG --- http response --- 10:24:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:24:10 DEBUG ----- output ----- 10:24:10 DEBUG { 10:24:10 DEBUG "status": "success", 10:24:10 DEBUG "data": { 10:24:10 DEBUG "resultType": "vector", 10:24:10 DEBUG "result": [ 10:24:10 DEBUG { 10:24:10 DEBUG "metric": { 10:24:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG [...] 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG 10:24:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:24:10 DEBUG }, 10:24:10 DEBUG "value": [ 10:24:10 DEBUG 1734776650, 10:24:10 DEBUG "0.04376857863100918" 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG ] 10:24:10 DEBUG } 10:24:10 DEBUG } 10:24:54 INFO 10:24:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:24:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:24:54 INFO [loop_until]: OK (rc = 0) 10:24:54 DEBUG --- stdout --- 10:24:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13709Mi rcs-5d696ff5b-24wr5 1m 348Mi rcs-5d696ff5b-vjklw 1m 842Mi 10:24:54 DEBUG --- stderr --- 10:24:54 DEBUG 10:24:59 INFO 10:24:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:24:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:24:59 INFO [loop_until]: OK (rc = 0) 10:24:59 DEBUG --- stdout --- 10:24:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 97m 0% 3170Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 92m 0% 19859Mi 69% 10:24:59 DEBUG --- stderr --- 10:24:59 DEBUG 10:25:09 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou-rcs%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776709" 10:25:09 INFO [http_cmd]: http status code OK 10:25:09 DEBUG --- status code --- 10:25:09 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:09 DEBUG --- http response --- 10:25:09 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou-rcs%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou-rcs%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_timed_waiting_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_memory_usage_pools_metaspace_usage%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_connection_handlers_ldap_active_connections_count%7Bldap_handler%3D%27LDAP%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_max_threads_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_replication_replica_replayed_updates_conflicts_resolved%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28ds_work_queue_requests_in_queue%7Bpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_blocked_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D%27amIdentityStore%27%2Cpod%3D~%27ds-idrepo.%2A%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou-rcs%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=max%28ds_disk_free_space_bytes%7Bdisk%3D%27/opt/opendj/db%27%2Cnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_scripting_threadpool_rejected_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28idm_managed_seconds%7Bnamespace%3D%27xlou-rcs%27%2Cmanaged_object%3D~%27.%2Auser.%2A%27%2Coperation%3D~%27%28create%7Cpatch%7Cupdate%29%27%2Cquantile%3D%270.5%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28am_jvm_thread_state_count%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 ERROR [http_cmd]: ERROR 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou-rcs%27%7D%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 ERROR --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- status code --- 10:25:10 ERROR http status code is 400 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776710" 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 ERROR --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 10:25:10 DEBUG --- http response --- 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG { 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "data": { 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG "data": { 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG { 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG "result": [ 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "result": [ 10:25:10 DEBUG { 10:25:10 DEBUG "data": { 10:25:10 DEBUG { 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "pod": "ds-idrepo-0" 10:25:10 DEBUG "result": [ 10:25:10 DEBUG "pod": "ds-cts-0" 10:25:10 DEBUG }, 10:25:10 DEBUG { 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG "instance": "10.142.0.11:9100" 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG }, 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG [...] 10:25:10 DEBUG 10:25:10 DEBUG [...] 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:25:10 DEBUG 10:25:10 DEBUG [...] 10:25:10 DEBUG }, 10:25:10 DEBUG "pod": "rcs-5d696ff5b-24wr5" 10:25:10 DEBUG 10:25:10 DEBUG "value": [ 10:25:10 DEBUG }, 10:25:10 DEBUG 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG "271.5460090161761" 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG "instance": "10.142.0.51:9100" 10:25:10 DEBUG ] 10:25:10 DEBUG "732127232" 10:25:10 DEBUG }, 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG "value": [ 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG "85421.70000000003" 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG { 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "data": { 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG "result": [ 10:25:10 DEBUG { 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "instance": "10.142.0.11:9100" 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG [...] 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG "instance": "10.142.0.51:9100" 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG "69203.53333333334" 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG { 10:25:10 DEBUG { 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "data": { 10:25:10 DEBUG "data": { 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG "result": [ 10:25:10 DEBUG "result": [ 10:25:10 DEBUG { 10:25:10 DEBUG { 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "pod": "ds-idrepo-0" 10:25:10 DEBUG "instance": "10.142.0.11:9100" 10:25:10 DEBUG }, 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG [...] 10:25:10 DEBUG [...] 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:25:10 DEBUG "instance": "10.142.0.51:9100" 10:25:10 DEBUG }, 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG "0" 10:25:10 DEBUG "0.0003333333333330302" 10:25:10 DEBUG ] 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776710" 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1734776710" 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou-rcs%27%7D%5B60s%5D%29%29by%28pod%29&time=1734776710" 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG { 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "data": { 10:25:10 DEBUG "resultType": "vector", 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG "result": [ 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG { 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG "metric": { 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1734776710" 10:25:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG }, 10:25:10 DEBUG { 10:25:10 DEBUG "value": [ 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG 10:25:10 DEBUG "data": { 10:25:10 DEBUG 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG 10:25:10 DEBUG "result": [ 10:25:10 DEBUG [...] 10:25:10 DEBUG { 10:25:10 DEBUG 10:25:10 DEBUG "metric": { 10:25:10 DEBUG 10:25:10 DEBUG "instance": "10.142.0.11:9100" 10:25:10 DEBUG 10:25:10 DEBUG }, 10:25:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:25:10 DEBUG "value": [ 10:25:10 DEBUG }, 10:25:10 DEBUG 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG 10:25:10 DEBUG "15778267136" 10:25:10 DEBUG [...] 10:25:10 DEBUG ] 10:25:10 DEBUG 10:25:10 DEBUG } 10:25:10 DEBUG 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG ] 10:25:10 DEBUG 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG } 10:25:10 DEBUG "instance": "10.142.0.51:9100" 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG } 10:25:10 DEBUG }, 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG "value": [ 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG "0.000466666666664158" 10:25:10 DEBUG { 10:25:10 DEBUG ] 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG } 10:25:10 DEBUG "data": { 10:25:10 DEBUG ] 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG } 10:25:10 DEBUG "result": [ 10:25:10 DEBUG } 10:25:10 DEBUG { 10:25:10 DEBUG "metric": { 10:25:10 DEBUG "instance": "10.142.0.11:9100" 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG [...] 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG 10:25:10 DEBUG "instance": "10.142.0.51:9100" 10:25:10 DEBUG }, 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG "24985.600000000002" 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG { 10:25:10 DEBUG "status": "success", 10:25:10 INFO [http_cmd]: http status code OK 10:25:10 DEBUG "data": { 10:25:10 DEBUG --- status code --- 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 10:25:10 DEBUG "result": [ 10:25:10 DEBUG --- http response --- 10:25:10 DEBUG { 10:25:10 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 10:25:10 DEBUG "metric": { 10:25:10 DEBUG ----- output ----- 10:25:10 DEBUG "pod": "ds-idrepo-0" 10:25:10 DEBUG { 10:25:10 DEBUG }, 10:25:10 DEBUG "status": "success", 10:25:10 DEBUG "value": [ 10:25:10 DEBUG "data": { 10:25:10 DEBUG 10:25:10 DEBUG "resultType": "vector", 10:25:10 DEBUG 10:25:10 DEBUG "result": [ 10:25:10 DEBUG 10:25:10 DEBUG { 10:25:10 DEBUG [...] 10:25:10 DEBUG "metric": { 10:25:10 DEBUG 10:25:10 DEBUG "node": "gke-xlou-cdm-default-pool-9d1e5395-7hzu" 10:25:10 DEBUG 10:25:10 DEBUG }, 10:25:10 DEBUG 10:25:10 DEBUG "value": [ 10:25:10 DEBUG "pod": "rcs-5d696ff5b-vjklw" 10:25:10 DEBUG 10:25:10 DEBUG }, 10:25:10 DEBUG 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG [...] 10:25:10 DEBUG "0.000584327764520645" 10:25:10 DEBUG 10:25:10 DEBUG ] 10:25:10 DEBUG 10:25:10 DEBUG } 10:25:10 DEBUG 10:25:10 DEBUG ] 10:25:10 DEBUG "node": "gke-xlou-cdm-frontend-18a44ad7-wxs9" 10:25:10 DEBUG } 10:25:10 DEBUG }, 10:25:10 DEBUG } 10:25:10 DEBUG "value": [ 10:25:10 DEBUG 1734776710, 10:25:10 DEBUG "0.045436152606890476" 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG ] 10:25:10 DEBUG } 10:25:10 DEBUG } 10:25:52 WARNING File /tmp/lodemon_data-20241220_182536/node_disk_read_bytes_total_file.json does not exist. Skipping average value computation... 127.0.0.1 - - [21/Dec/2024 10:25:52] "GET /monitoring/average?start_time=20241220_182849&stop_time=20241221_102351 HTTP/1.1" 200 - 10:25:54 INFO 10:25:54 INFO [loop_until]: kubectl --namespace=xlou-rcs top pods 10:25:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:25:54 INFO [loop_until]: OK (rc = 0) 10:25:54 DEBUG --- stdout --- 10:25:54 DEBUG NAME CPU(cores) MEMORY(bytes) ds-cts-0 4m 525Mi ds-idrepo-0 6m 13708Mi rcs-5d696ff5b-24wr5 2m 348Mi rcs-5d696ff5b-vjklw 2m 842Mi 10:25:54 DEBUG --- stderr --- 10:25:54 DEBUG 10:25:59 INFO 10:25:59 INFO [loop_until]: kubectl --namespace=xlou-rcs top node 10:25:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 10:25:59 INFO [loop_until]: OK (rc = 0) 10:25:59 DEBUG --- stdout --- 10:25:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-9d1e5395-7hzu 96m 0% 3172Mi 5% gke-xlou-cdm-frontend-18a44ad7-wxs9 99m 0% 19859Mi 69% 10:25:59 DEBUG --- stderr --- 10:25:59 DEBUG 10:26:05 INFO Finished: True 10:26:05 INFO Waiting for threads to register finish flag 10:26:59 INFO Done. Have a nice day! :) 127.0.0.1 - - [21/Dec/2024 10:26:59] "GET /monitoring/stop HTTP/1.1" 200 - 10:27:01 WARNING File /tmp/lodemon_data-20241220_182536/node_disk_read_bytes_total_file.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3254', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3136', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2989', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2205', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2891', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3038', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2450', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3450', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2548', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2401', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2499', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3401', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2793', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2352', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3352', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3303', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p3087', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2646', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2842', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2303', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2940', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2695', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2744', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2254', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2597', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p2156', ...) 127.0.0.1 - - [21/Dec/2024 10:27:03] "GET /monitoring/process HTTP/1.1" 200 -