==================================================================================================== ============================ Pod describe lodemon-67b8bd6fff-l8gt5 ============================ ==================================================================================================== Name: lodemon-67b8bd6fff-l8gt5 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Mon, 15 Sep 2025 17:32:27 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=67b8bd6fff skaffold.dev/run-id=3f108aaa-952d-4084-b662-7092a740091d Annotations: Status: Running IP: 10.106.40.157 IPs: IP: 10.106.40.157 Controlled By: ReplicaSet/lodemon-67b8bd6fff Containers: lodemon: Container ID: containerd://502503e1ca80c0e617ecbe64168739bc6cc43ee5aa324ff4193ea5ed863e0df3 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Mon, 15 Sep 2025 17:32:28 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tn56n (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-tn56n: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-67b8bd6fff-l8gt5 ============================== ==================================================================================================== 2025-09-15 17:32:28 INF Starting lodemon server on port 8080 2025-09-15 17:32:28 INF Initializing clients... 2025-09-15 17:32:28 DBG Fetching ID token for accessing prometheus 2025-09-15 17:32:28 DBG Fetched ID token successfully 2025-09-15 17:32:28 INF Clients initialized successfully 2025-09-15 17:33:03 INF Configuring Lodemon 2025-09-15 17:33:03 INF Configuring monitoring units 2025-09-15 17:33:03 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-15 17:33:03 TRC Parsing Prometheus queries from config 2025-09-15 17:33:03 TRC Parsing component query: DS Java heap mem usage 2025-09-15 17:33:03 TRC Parsing component query: IDM Java heap mem usage 2025-09-15 17:33:03 TRC Parsing component query: AM Java heap mem usage 2025-09-15 17:33:03 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-15 17:33:03 TRC Parsing global query: 2025-09-15 17:33:03 TRC Parsing scoped query: DS Identity backend connections 2025-09-15 17:33:03 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-15 17:33:03 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-15 17:33:03 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-15 17:33:03 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-15 17:33:03 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-15 17:33:03 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-15 17:33:03 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-15 17:33:03 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-15 17:33:03 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-15 17:33:03 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-15 17:33:03 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-15 17:33:03 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-15 17:33:03 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-15 17:33:03 INF  - All monitoring units configured 2025-09-15 17:33:03 INF Configuring monitoring service 2025-09-15 17:33:03 INF  - Adding monitoring units to the service 2025-09-15 17:33:03 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-15 17:33:03 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-15 17:33:03 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-15 17:33:03 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-15 17:33:03 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-15 17:33:03 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-15 17:33:03 INF  - All monitoring units added to the service 2025-09-15 17:33:03 INF Starting Lodemon now 2025-09-15 17:33:03 INF Starting monitoring service... 2025-09-15 17:33:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:33:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:33:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:33:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:33:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:33:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:33:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:33:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:33:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:33:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:33:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:33:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:33:09 INF Adding event: Scenario_benchmark 2025-09-15 17:33:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:33:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:33:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:33:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:33:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:33:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:33:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:33:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:33:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:33:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:33:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:33:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:33:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:33:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:33:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:33:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:33:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:33:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:33:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:33:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:33:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:33:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:33:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:33:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:33:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:33:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:33:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:33:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:33:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:33:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:33:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:33:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:33:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:33:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:33:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:33:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:33:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:33:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:33:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:33:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:33:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:33:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:33:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:33:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:33:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:33:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:33:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:33:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:33:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:33:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:33:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:33:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:33:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:33:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:33:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:33:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:33:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:33:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:33:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:33:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:33:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:33:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:34:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:34:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:34:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:34:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:34:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:34:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:34:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:34:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:34:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:34:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:34:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:34:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:34:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:34:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:34:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:34:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:34:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:34:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:34:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:34:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:34:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:34:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:34:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:34:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:34:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:34:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:34:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:34:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:34:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:34:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:34:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:34:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:34:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:34:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:34:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:34:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:34:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:34:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:34:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:34:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:34:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:34:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:34:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:34:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:34:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:34:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:34:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:34:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:34:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:34:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:34:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:34:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:34:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:34:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:34:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:34:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:34:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:34:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:34:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:34:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:34:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:34:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:34:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:34:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:34:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:34:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:34:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:34:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:34:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:34:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:34:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:34:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:34:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:34:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:35:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:35:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:35:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:35:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:35:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:35:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:35:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:35:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:35:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:35:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:35:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:35:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:35:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:35:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:35:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:35:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:35:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:35:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:35:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:35:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:35:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:35:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:35:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:35:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:35:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:35:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:35:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:35:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:35:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:35:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:35:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:35:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:35:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:35:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:35:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:35:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:35:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:35:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:35:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:35:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:35:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:35:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:35:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:35:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:35:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:35:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:35:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:35:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:35:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:35:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:35:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:35:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:35:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:35:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:35:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:35:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:35:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:35:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:35:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:35:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:35:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:35:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:35:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:35:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:35:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:35:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:35:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:35:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:35:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:36:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:36:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:36:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:36:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:36:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:36:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:36:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:36:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:36:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:36:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:36:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:36:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:36:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:36:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:36:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:36:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:36:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:36:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:36:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:36:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:36:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:36:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:36:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:36:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:36:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:36:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:36:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:36:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:36:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:36:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:36:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:36:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:36:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:36:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:36:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:36:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:36:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:36:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:36:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:36:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:36:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:36:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:36:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:36:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:36:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:36:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:36:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:36:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:36:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:36:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:36:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:36:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:36:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:36:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:36:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:36:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:36:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:36:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:36:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:36:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:36:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:36:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:36:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:36:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:36:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:36:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:36:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:36:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:36:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:36:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:36:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:36:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:36:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:36:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:37:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:37:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:37:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:37:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:37:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:37:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:37:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:37:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:37:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:37:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:37:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:37:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:37:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:37:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:37:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:37:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:37:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:37:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:37:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:37:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:37:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:37:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:37:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:37:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:37:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:37:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:37:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:37:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:37:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:37:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:37:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:37:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:37:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:37:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:37:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:37:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:37:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:37:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:37:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:37:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:37:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:37:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:37:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:37:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:37:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:37:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:37:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:37:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:37:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:37:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:37:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:37:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:37:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:37:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:37:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:37:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:37:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:37:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:37:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:37:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:37:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:37:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:37:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:37:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:37:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:37:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:37:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:37:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:37:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:37:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:37:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:37:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:37:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:37:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:38:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:38:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:38:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:38:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:38:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:38:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:38:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:38:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:38:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:38:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:38:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:38:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:38:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:38:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:38:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:38:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:38:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:38:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:38:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:38:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:38:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:38:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:38:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:38:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:38:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:38:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:38:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:38:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:38:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:38:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:38:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:38:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:38:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:38:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:38:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:38:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:38:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:38:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:38:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:38:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:38:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:38:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:38:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:38:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:38:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:38:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:38:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:38:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:38:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:38:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:38:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:38:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:38:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:38:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:38:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:38:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:38:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:38:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:38:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:38:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:38:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:38:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:38:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:38:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:38:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:38:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:38:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:38:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:38:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:38:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:38:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:38:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:38:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:38:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:39:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:39:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:39:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:39:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:39:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:39:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:39:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:39:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:39:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:39:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:39:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:39:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:39:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:39:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:39:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:39:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:39:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:39:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:39:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:39:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:39:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:39:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:39:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:39:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:39:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:39:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:39:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:39:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:39:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:39:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:39:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:39:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:39:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:39:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:39:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:39:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:39:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:39:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:39:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:39:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:39:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:39:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:39:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:39:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:39:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:39:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:39:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:39:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:39:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:39:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:39:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:39:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:39:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:39:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:39:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:39:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:39:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:39:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:39:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:39:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:39:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:39:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:39:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:39:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:39:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:39:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:39:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:39:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:39:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:39:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:39:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:39:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:39:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:39:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:40:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:40:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:40:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:40:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:40:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:40:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:40:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:40:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:40:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:40:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:40:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:40:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:40:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:40:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:40:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:40:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:40:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:40:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:40:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:40:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:40:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:40:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:40:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:40:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:40:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:40:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:40:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:40:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:40:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:40:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:40:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:40:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:40:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:40:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:40:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:40:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:40:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:40:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:40:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:40:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:40:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:40:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:40:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:40:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:40:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:40:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:40:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:40:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:40:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:40:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:40:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:40:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:40:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:40:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:40:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:40:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:40:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:40:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:40:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:40:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:40:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:40:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:40:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:40:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:40:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:40:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:40:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:40:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:40:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:40:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:40:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:40:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:40:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:40:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:41:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:41:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:41:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:41:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:41:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:41:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:41:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:41:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:41:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:41:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:41:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:41:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:41:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:41:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:41:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:41:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:41:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:41:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:41:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:41:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:41:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:41:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:41:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:41:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:41:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:41:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:41:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:41:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:41:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:41:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:41:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:41:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:41:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:41:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:41:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:41:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:41:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:41:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:41:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:41:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:41:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:41:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:41:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:41:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:41:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:41:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:41:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:41:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:41:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:41:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:41:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:41:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:41:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:41:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:41:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:41:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:41:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:41:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:41:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:41:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:41:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:41:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:41:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:41:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:41:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:41:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:41:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:41:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:41:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:41:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:41:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:41:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:41:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:41:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:42:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:42:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:42:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:42:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:42:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:42:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:42:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:42:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:42:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:42:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:42:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:42:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:42:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:42:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:42:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:42:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:42:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:42:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:42:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:42:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:42:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:42:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:42:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:42:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:42:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:42:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:42:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:42:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:42:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:42:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:42:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:42:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:42:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:42:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:42:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:42:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:42:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:42:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:42:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:42:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:42:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:42:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:42:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:42:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:42:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:42:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:42:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:42:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:42:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:42:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:42:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:42:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:42:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:42:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:42:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:42:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:42:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:42:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:42:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:42:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:42:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:43:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:43:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:43:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:43:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:43:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:43:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:43:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:43:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:43:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:43:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:43:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:43:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:43:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:43:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:43:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:43:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:43:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:43:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:43:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:43:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:43:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:43:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:43:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:43:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:43:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:43:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:43:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:43:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:43:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:43:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:43:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:43:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:43:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:43:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:43:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:43:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:43:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:43:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:43:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:43:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:43:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:43:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:43:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:43:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:43:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:43:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:43:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:43:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:43:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:43:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:43:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:43:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:43:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:43:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:43:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:43:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:43:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:43:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:43:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:43:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:43:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:43:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:43:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:43:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:43:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:43:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:43:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:43:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:43:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:43:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:43:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:43:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:43:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:43:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:44:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:44:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:44:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:44:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:44:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:44:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:44:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:44:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:44:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:44:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:44:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:44:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:44:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:44:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:44:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:44:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:44:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:44:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:44:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:44:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:44:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:44:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:44:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:44:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:44:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:44:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:44:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:44:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:44:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:44:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:44:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:44:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:44:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:44:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:44:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:44:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:44:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:44:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:44:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:44:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:44:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:44:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:44:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:44:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:44:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:44:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:44:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:44:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:44:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:44:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:44:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:44:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:44:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:44:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:44:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:44:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:44:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:44:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:44:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:44:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:44:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:44:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:44:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:44:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:44:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:44:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:44:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:44:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:44:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:45:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:45:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:45:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:45:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:45:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:45:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:45:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:45:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:45:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:45:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:45:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:45:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:45:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:45:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:45:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:45:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:45:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:45:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:45:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:45:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:45:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:45:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:45:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:45:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:45:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:45:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:45:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:45:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:45:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:45:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:45:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:45:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:45:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:45:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:45:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:45:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:45:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:45:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:45:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:45:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:45:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:45:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:45:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:45:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:45:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:45:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:45:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:45:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:45:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:45:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:45:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:45:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:45:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:45:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:45:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:45:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:45:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:45:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:45:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:45:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:45:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:45:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:45:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:45:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:45:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:45:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:45:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:45:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:45:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:45:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:45:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:45:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:45:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:46:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:46:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:46:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:46:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:46:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:46:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:46:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:46:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:46:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:46:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:46:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:46:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:46:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:46:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:46:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:46:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:46:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:46:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:46:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:46:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:46:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:46:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:46:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:46:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:46:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:46:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:46:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:46:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:46:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:46:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:46:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:46:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:46:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:46:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:46:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:46:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:46:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:46:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:46:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:46:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:46:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:46:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:46:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:46:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:46:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:46:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:46:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:46:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:46:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:46:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:46:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:46:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:46:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:46:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:46:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:46:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:46:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:46:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:46:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:46:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:46:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:46:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:46:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:46:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:46:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:46:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:46:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:46:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:46:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:47:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:47:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:47:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:47:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:47:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:47:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:47:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:47:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:47:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:47:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:47:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:47:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:47:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:47:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:47:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:47:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:47:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:47:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:47:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:47:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:47:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:47:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:47:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:47:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:47:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:47:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:47:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:47:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:47:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:47:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:47:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:47:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:47:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:47:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:47:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:47:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:47:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:47:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:47:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:47:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:47:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:47:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:47:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:47:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:47:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:47:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:47:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:47:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:47:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:47:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:47:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:47:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:47:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:47:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:47:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:47:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:47:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:47:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:47:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:47:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:47:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:47:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:47:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:47:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:47:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:47:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:47:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:47:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:47:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:47:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:47:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:47:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:47:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:48:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:48:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:48:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:48:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:48:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:48:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:48:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:48:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:48:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:48:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:48:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:48:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:48:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:48:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:48:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:48:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:48:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:48:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:48:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:48:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:48:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:48:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:48:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:48:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:48:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:48:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:48:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:48:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:48:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:48:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:48:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:48:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:48:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:48:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:48:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:48:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:48:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:48:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:48:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:48:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:48:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:48:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:48:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:48:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:48:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:48:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:48:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:48:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:48:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:48:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:48:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:48:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:48:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:48:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:48:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:48:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:48:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:48:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:48:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:48:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:48:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:48:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:48:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:48:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:48:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:48:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:48:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:48:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:48:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:49:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:49:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:49:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:49:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:49:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:49:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:49:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:49:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:49:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:49:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:49:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:49:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:49:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:49:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:49:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:49:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:49:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:49:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:49:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:49:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:49:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:49:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:49:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:49:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:49:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:49:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:49:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:49:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:49:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:49:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:49:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:49:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:49:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:49:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:49:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:49:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:49:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:49:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:49:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:49:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:49:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:49:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:49:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:49:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:49:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:49:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:49:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:49:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:49:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:49:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:49:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:49:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:49:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:49:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:49:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:49:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:49:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:49:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:49:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:49:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:49:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:49:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:49:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:49:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:49:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:49:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:49:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:49:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:49:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:49:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:49:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:49:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:49:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:50:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:50:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:50:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:50:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:50:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:50:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:50:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:50:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:50:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:50:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:50:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:50:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:50:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:50:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:50:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:50:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:50:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:50:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:50:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:50:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:50:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:50:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:50:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:50:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:50:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:50:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:50:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:50:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:50:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:50:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:50:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:50:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:50:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:50:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:50:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:50:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:50:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:50:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:50:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:50:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:50:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:50:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:50:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:50:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:50:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:50:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:50:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:50:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:50:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:50:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:50:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:50:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:50:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:50:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:50:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:50:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:50:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:50:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:50:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:50:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:50:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:50:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:50:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:50:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:50:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:50:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:50:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:50:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:51:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:51:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:51:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:51:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:51:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:51:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:51:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:51:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:51:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:51:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:51:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:51:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:51:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:51:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:51:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:51:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:51:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:51:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:51:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:51:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:51:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:51:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:51:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:51:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:51:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:51:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:51:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:51:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:51:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:51:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:51:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:51:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:51:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:51:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:51:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:51:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:51:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:51:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:51:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:51:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:51:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:51:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:51:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:51:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:51:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:51:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:51:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:51:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:51:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:51:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:51:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:51:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:51:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:51:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:51:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:51:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:51:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:51:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:51:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:51:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:51:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:51:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:51:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:51:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:51:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:51:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:51:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:51:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:51:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:51:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:51:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:51:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:51:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:51:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:52:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:52:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:52:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:52:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:52:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:52:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:52:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:52:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:52:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:52:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:52:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:52:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:52:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:52:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:52:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:52:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:52:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:52:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:52:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:52:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:52:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:52:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:52:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:52:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:52:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:52:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:52:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:52:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:52:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:52:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:52:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:52:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:52:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:52:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:52:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:52:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:52:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:52:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:52:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:52:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:52:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:52:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:52:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:52:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:52:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:52:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:52:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:52:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:52:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:52:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:52:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:52:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:52:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:52:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:52:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:52:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:52:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:52:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:52:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:52:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:52:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:52:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:52:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:52:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:52:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:52:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:52:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:52:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:52:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:53:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:53:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:53:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:53:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:53:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:53:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:53:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:53:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:53:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:53:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:53:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:53:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:53:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:53:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:53:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:53:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:53:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:53:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:53:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:53:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:53:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:53:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:53:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:53:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:53:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:53:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:53:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:53:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:53:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:53:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:53:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:53:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:53:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:53:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:53:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:53:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:53:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:53:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:53:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:53:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:53:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:53:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:53:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:53:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:53:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:53:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:53:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:53:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:53:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:53:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:53:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:53:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:53:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:53:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:53:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:53:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:53:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:53:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:53:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:53:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:53:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:53:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:53:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:53:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:53:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:53:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:53:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:53:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:53:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:53:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:53:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:53:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:53:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:54:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:54:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:54:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:54:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:54:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:54:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:54:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:54:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:54:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:54:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:54:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:54:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:54:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:54:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:54:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:54:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:54:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:54:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:54:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:54:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:54:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:54:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:54:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:54:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:54:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:54:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:54:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:54:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:54:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:54:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:54:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:54:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:54:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:54:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:54:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:54:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:54:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:54:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:54:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:54:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:54:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:54:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:54:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:54:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:54:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:54:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:54:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:54:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:54:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:54:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:54:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:54:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:54:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:54:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:54:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:54:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:54:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:54:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:54:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:54:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:54:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:54:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:54:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:54:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:54:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:54:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:54:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:54:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:55:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:55:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:55:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:55:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:55:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:55:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:55:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:55:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:55:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:55:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:55:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:55:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:55:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:55:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:55:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:55:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:55:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:55:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:55:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:55:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:55:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:55:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:55:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:55:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:55:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:55:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:55:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:55:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:55:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:55:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:55:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:55:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:55:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:55:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:55:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:55:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:55:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:55:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:55:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:55:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:55:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:55:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:55:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:55:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:55:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:55:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:55:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:55:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:55:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:55:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:55:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:55:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:55:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:55:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:55:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:55:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:55:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:55:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:55:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:55:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:55:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:55:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:55:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:56:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:56:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:56:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:56:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:56:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:56:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:56:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:56:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:56:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:56:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:56:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:56:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:56:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:56:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:56:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:56:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:56:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:56:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:56:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:56:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:56:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:56:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:56:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:56:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:56:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:56:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:56:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:56:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:56:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:56:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:56:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:56:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:56:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:56:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:56:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:56:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:56:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:56:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:56:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:56:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:56:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:56:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:56:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:56:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:56:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:56:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:56:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:56:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:56:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:56:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:56:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:56:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:56:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:56:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:56:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:56:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:56:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:56:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:56:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:56:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:56:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:56:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:56:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:56:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:56:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:56:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:56:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:56:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:56:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:56:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:56:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:56:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:56:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:56:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:57:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:57:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:57:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:57:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:57:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:57:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:57:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:57:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:57:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:57:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:57:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:57:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:57:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:57:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:57:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:57:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:57:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:57:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:57:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:57:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:57:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:57:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:57:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:57:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:57:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:57:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:57:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:57:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:57:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:57:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:57:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:57:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:57:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:57:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:57:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:57:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:57:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:57:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:57:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:57:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:57:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:57:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:57:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:57:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:57:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:57:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:57:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:57:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:57:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:57:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:57:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:57:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:57:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:57:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:57:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:57:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:57:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:57:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:57:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:57:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:57:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:57:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:57:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:57:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:57:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:57:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:57:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:57:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:57:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:57:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:57:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:57:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:57:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:58:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:58:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:58:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:58:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:58:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:58:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:58:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:58:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:58:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:58:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:58:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:58:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:58:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:58:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:58:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:58:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:58:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:58:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:58:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:58:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:58:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:58:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:58:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:58:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:58:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:58:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:58:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:58:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:58:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:58:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:58:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:58:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:58:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:58:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:58:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:58:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:58:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:58:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:58:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:58:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:58:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:58:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:58:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:58:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:58:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:58:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:58:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:58:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:58:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:58:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:58:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:58:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:58:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:58:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:58:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:58:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:58:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:58:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:58:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:58:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:58:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:58:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:58:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:58:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:58:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:58:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:58:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:58:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:58:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:59:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:59:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:59:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:59:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:59:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:59:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:59:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:59:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:59:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:59:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:59:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:59:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:59:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:59:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:59:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:59:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:59:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:59:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:59:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:59:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:59:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:59:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:59:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:59:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:59:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:59:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:59:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:59:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:59:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:59:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:59:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:59:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:59:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:59:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:59:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:59:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:59:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:59:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:59:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:59:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:59:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:59:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:59:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:59:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:59:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:59:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:59:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:59:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:59:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:59:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:59:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:59:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:59:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:59:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:59:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:59:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:59:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:59:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:59:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:59:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 17:59:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 17:59:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 17:59:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 17:59:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 17:59:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 17:59:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 17:59:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 17:59:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 17:59:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 17:59:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 17:59:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 17:59:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 17:59:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 17:59:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:00:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:00:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:00:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:00:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:00:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:00:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:00:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:00:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:00:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:00:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:00:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:00:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:00:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:00:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:00:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:00:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:00:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:00:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:00:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:00:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:00:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:00:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:00:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:00:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:00:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:00:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:00:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:00:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:00:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:00:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:00:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:00:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:00:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:00:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:00:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:00:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:00:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:00:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:00:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:00:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:00:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:00:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:00:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:00:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:00:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:00:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:00:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:00:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:00:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:00:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:00:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:00:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:00:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:00:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:00:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:00:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:00:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:00:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:00:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:00:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:00:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:00:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:00:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:00:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:00:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:00:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:00:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:00:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:00:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:00:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:00:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:00:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:00:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:00:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:01:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:01:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:01:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:01:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:01:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:01:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:01:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:01:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:01:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:01:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:01:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:01:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:01:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:01:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:01:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:01:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:01:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:01:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:01:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:01:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:01:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:01:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:01:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:01:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:01:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:01:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:01:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:01:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:01:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:01:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:01:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:01:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:01:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:01:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:01:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:01:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:01:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:01:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:01:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:01:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:01:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:01:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:01:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:01:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:01:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:01:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:01:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:01:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:01:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:01:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:01:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:01:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:01:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:01:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:01:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:01:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:01:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:01:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:01:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:01:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:01:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:01:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:01:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:01:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:01:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:01:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:01:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:01:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:01:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:01:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:01:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:01:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:01:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:02:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:02:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:02:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:02:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:02:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:02:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:02:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:02:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:02:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:02:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:02:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:02:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:02:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:02:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:02:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:02:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:02:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:02:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:02:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:02:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:02:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:02:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:02:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:02:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:02:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:02:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:02:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:02:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:02:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:02:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:02:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:02:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:02:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:02:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:02:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:02:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:02:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:02:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:02:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:02:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:02:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:02:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:02:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:02:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:02:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:02:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:02:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:02:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:02:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:02:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:02:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:02:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:02:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:02:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:02:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:02:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:02:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:02:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:02:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:02:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:02:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:02:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:02:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:02:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:02:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:02:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:02:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:02:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:02:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:03:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:03:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:03:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:03:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:03:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:03:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:03:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:03:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:03:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:03:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:03:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:03:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:03:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:03:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:03:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:03:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:03:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:03:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:03:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:03:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:03:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:03:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:03:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:03:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:03:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:03:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:03:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:03:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:03:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:03:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:03:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:03:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:03:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:03:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:03:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:03:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:03:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:03:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:03:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:03:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:03:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:03:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:03:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:03:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:03:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:03:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:03:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:03:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:03:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:03:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:03:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:03:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:03:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:03:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:03:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:03:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:03:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:03:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:03:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:03:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:03:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:03:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:03:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:03:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:03:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:03:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:03:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:03:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:03:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:03:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:03:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:03:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:03:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:03:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:04:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:04:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:04:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:04:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:04:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:04:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:04:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:04:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:04:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:04:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:04:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:04:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:04:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:04:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:04:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:04:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:04:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:04:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:04:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:04:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:04:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:04:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:04:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:04:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:04:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:04:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:04:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:04:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:04:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:04:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:04:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:04:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:04:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:04:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:04:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:04:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:04:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:04:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:04:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:04:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:04:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:04:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:04:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:04:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:04:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:04:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:04:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:04:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:04:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:04:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:04:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:04:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:04:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:04:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:04:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:04:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:04:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:04:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:04:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:04:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:04:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:04:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:04:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:04:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:04:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:04:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:04:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:04:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:04:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:04:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:04:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:04:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:04:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:04:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:05:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:05:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:05:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:05:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:05:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:05:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:05:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:05:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:05:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:05:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:05:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:05:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:05:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:05:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:05:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:05:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:05:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:05:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:05:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:05:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:05:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:05:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:05:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:05:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:05:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:05:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:05:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:05:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:05:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:05:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:05:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:05:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:05:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:05:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:05:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:05:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:05:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:05:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:05:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:05:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:05:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:05:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:05:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:05:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:05:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:05:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:05:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:05:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:05:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:05:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:05:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:05:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:05:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:05:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:05:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:05:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:05:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:05:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:05:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:05:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:05:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:05:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:05:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:05:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:05:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:05:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:05:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:05:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:05:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:05:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:05:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:05:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:05:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:06:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:06:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:06:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:06:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:06:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:06:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:06:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:06:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:06:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:06:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:06:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:06:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:06:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:06:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:06:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:06:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:06:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:06:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:06:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:06:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:06:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:06:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:06:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:06:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:06:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:06:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:06:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:06:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:06:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:06:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:06:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:06:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:06:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:06:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:06:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:06:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:06:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:06:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:06:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:06:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:06:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:06:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:06:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:06:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:06:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:06:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:06:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:06:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:06:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:06:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:06:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:06:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:06:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:06:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:06:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:06:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:06:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:06:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:06:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:06:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:06:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:06:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:06:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:06:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:06:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:06:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:06:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:06:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:06:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:06:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:06:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:06:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:06:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:06:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:07:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:07:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:07:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:07:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:07:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:07:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:07:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:07:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:07:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:07:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:07:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:07:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:07:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:07:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:07:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:07:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:07:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:07:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:07:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:07:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:07:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:07:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:07:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:07:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:07:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:07:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:07:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:07:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:07:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:07:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:07:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:07:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:07:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:07:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:07:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:07:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:07:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:07:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:07:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:07:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:07:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:07:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:07:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:07:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:07:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:07:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:07:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:07:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:07:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:07:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:07:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:07:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:07:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:07:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:07:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:07:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:07:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:07:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:07:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:07:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:07:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:07:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:07:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:07:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:07:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:07:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:07:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:07:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:08:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:08:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:08:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:08:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:08:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:08:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:08:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:08:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:08:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:08:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:08:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:08:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:08:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:08:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:08:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:08:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:08:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:08:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:08:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:08:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:08:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:08:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:08:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:08:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:08:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:08:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:08:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:08:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:08:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:08:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:08:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:08:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:08:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:08:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:08:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:08:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:08:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:08:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:08:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:08:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:08:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:08:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:08:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:08:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:08:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:08:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:08:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:08:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:08:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:08:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:08:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:08:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:08:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:08:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:08:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:08:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:08:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:08:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:08:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:08:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:08:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:08:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:09:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:09:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:09:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:09:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:09:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:09:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:09:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:09:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:09:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:09:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:09:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:09:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:09:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:09:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:09:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:09:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:09:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:09:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:09:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:09:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:09:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:09:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:09:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:09:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:09:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:09:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:09:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:09:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:09:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:09:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:09:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:09:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:09:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:09:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:09:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:09:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:09:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:09:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:09:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:09:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:09:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:09:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:09:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:09:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:09:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:09:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:09:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:09:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:09:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:09:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:09:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:09:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:09:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:09:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:09:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:09:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:09:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:09:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:09:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:09:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:09:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:09:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:09:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:09:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:09:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:09:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:09:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:09:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:09:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:10:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:10:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:10:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:10:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:10:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:10:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:10:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:10:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:10:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:10:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:10:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:10:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:10:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:10:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:10:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:10:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:10:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:10:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:10:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:10:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:10:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:10:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:10:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:10:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:10:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:10:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:10:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:10:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:10:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:10:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:10:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:10:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:10:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:10:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:10:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:10:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:10:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:10:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:10:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:10:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:10:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:10:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:10:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:10:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:10:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:10:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:10:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:10:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:10:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:10:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:10:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:10:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:10:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:10:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:10:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:10:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:10:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:10:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:10:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:10:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:10:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:10:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:10:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:10:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:10:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:10:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:10:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:10:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:10:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:10:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:10:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:10:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:10:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:10:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:11:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:11:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:11:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:11:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:11:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:11:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:11:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:11:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:11:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:11:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:11:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:11:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:11:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:11:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:11:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:11:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:11:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:11:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:11:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:11:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:11:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:11:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:11:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:11:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:11:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:11:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:11:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:11:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:11:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:11:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:11:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:11:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:11:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:11:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:11:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:11:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:11:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:11:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:11:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:11:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:11:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:11:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:11:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:11:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:11:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:11:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:11:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:11:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:11:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:11:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:11:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:11:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:11:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:11:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:11:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:11:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:11:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:11:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:11:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:11:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:11:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:11:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:11:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:11:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:11:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:11:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:11:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:11:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:11:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:11:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:11:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:11:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:11:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:12:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:12:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:12:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:12:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:12:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:12:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:12:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:12:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:12:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:12:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:12:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:12:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:12:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:12:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:12:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:12:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:12:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:12:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:12:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:12:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:12:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:12:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:12:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:12:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:12:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:12:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:12:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:12:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:12:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:12:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:12:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:12:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:12:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:12:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:12:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:12:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:12:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:12:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:12:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:12:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:12:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:12:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:12:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:12:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:12:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:12:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:12:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:12:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:12:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:12:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:12:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:12:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:12:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:12:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:12:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:12:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:12:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:12:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:12:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:12:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:12:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:12:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:12:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:12:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:12:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:12:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:12:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:12:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:13:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:13:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:13:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:13:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:13:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:13:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:13:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:13:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:13:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:13:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:13:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:13:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:13:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:13:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:13:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:13:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:13:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:13:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:13:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:13:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:13:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:13:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:13:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:13:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:13:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:13:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:13:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:13:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:13:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:13:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:13:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:13:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:13:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:13:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:13:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:13:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:13:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:13:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:13:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:13:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:13:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:13:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:13:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:13:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:13:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:13:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:13:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:13:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:13:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:13:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:13:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:13:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:13:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:13:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:13:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:13:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:13:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:13:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:13:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:13:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:13:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:13:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:13:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:13:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:13:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:13:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:13:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:13:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:13:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:13:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:13:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:13:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:13:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:13:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:14:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:14:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:14:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:14:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:14:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:14:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:14:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:14:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:14:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:14:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:14:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:14:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:14:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:14:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:14:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:14:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:14:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:14:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:14:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:14:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:14:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:14:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:14:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:14:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:14:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:14:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:14:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:14:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:14:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:14:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:14:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:14:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:14:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:14:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:14:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:14:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:14:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:14:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:14:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:14:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:14:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:14:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:14:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:14:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:14:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:14:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:14:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:14:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:14:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:14:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:14:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:14:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:14:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:14:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:14:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:14:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:14:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:14:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:14:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:14:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:14:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:14:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:14:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:14:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:14:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:14:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:14:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:14:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:14:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:15:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:15:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:15:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:15:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:15:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:15:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:15:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:15:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:15:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:15:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:15:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:15:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:15:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:15:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:15:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:15:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:15:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:15:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:15:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:15:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:15:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:15:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:15:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:15:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:15:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:15:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:15:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:15:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:15:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:15:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:15:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:15:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:15:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:15:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:15:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:15:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:15:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:15:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:15:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:15:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:15:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:15:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:15:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:15:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:15:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:15:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:15:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:15:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:15:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:15:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:15:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:15:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:15:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:15:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:15:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:15:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:15:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:15:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:15:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:15:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:15:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:15:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:15:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:15:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:15:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:15:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:15:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:15:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:15:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:15:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:15:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:15:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:15:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:16:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:16:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:16:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:16:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:16:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:16:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:16:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:16:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:16:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:16:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:16:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:16:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:16:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:16:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:16:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:16:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:16:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:16:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:16:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:16:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:16:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:16:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:16:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:16:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:16:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:16:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:16:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:16:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:16:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:16:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:16:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:16:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:16:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:16:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:16:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:16:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:16:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:16:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:16:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:16:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:16:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:16:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:16:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:16:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:16:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:16:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:16:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:16:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:16:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:16:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:16:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:16:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:16:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:16:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:16:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:16:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:16:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:16:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:16:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:16:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:16:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:16:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:16:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:16:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:16:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:16:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:16:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:16:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:16:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:16:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:16:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:16:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:16:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:16:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:17:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:17:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:17:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:17:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:17:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:17:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:17:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:17:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:17:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:17:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:17:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:17:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:17:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:17:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:17:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:17:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:17:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:17:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:17:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:17:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:17:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:17:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:17:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:17:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:17:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:17:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:17:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:17:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:17:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:17:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:17:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:17:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:17:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:17:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:17:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:17:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:17:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:17:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:17:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:17:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:17:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:17:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:17:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:17:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:17:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:17:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:17:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:17:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:17:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:17:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:17:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:17:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:17:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:17:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:17:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:17:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:17:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:17:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:17:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:17:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:17:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:17:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:17:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:17:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:17:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:17:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:17:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:17:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:17:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:18:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:18:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:18:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:18:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:18:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:18:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:18:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:18:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:18:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:18:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:18:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:18:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:18:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:18:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:18:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:18:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:18:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:18:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:18:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:18:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:18:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:18:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:18:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:18:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:18:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:18:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:18:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:18:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:18:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:18:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:18:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:18:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:18:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:18:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:18:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:18:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:18:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:18:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:18:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:18:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:18:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:18:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:18:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:18:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:18:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:18:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:18:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:18:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:18:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:18:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:18:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:18:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:18:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:18:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:18:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:18:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:18:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:18:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:18:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:18:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:18:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:18:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:18:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:18:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:18:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:18:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:18:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:18:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:18:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:18:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:18:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:18:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:18:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:18:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:19:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:19:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:19:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:19:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:19:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:19:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:19:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:19:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:19:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:19:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:19:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:19:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:19:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:19:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:19:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:19:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:19:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:19:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:19:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:19:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:19:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:19:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:19:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:19:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:19:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:19:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:19:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:19:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:19:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:19:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:19:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:19:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:19:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:19:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:19:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:19:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:19:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:19:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:19:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:19:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:19:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:19:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:19:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:19:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:19:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:19:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:19:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:19:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:19:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:19:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:19:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:19:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:19:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:19:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:19:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:19:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:19:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:19:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:19:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:19:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:19:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:19:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:19:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:19:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:19:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:19:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:19:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:19:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:19:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:19:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:19:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:19:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:19:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:20:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:20:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:20:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:20:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:20:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:20:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:20:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:20:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:20:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:20:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:20:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:20:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:20:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:20:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:20:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:20:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:20:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:20:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:20:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:20:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:20:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:20:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:20:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:20:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:20:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:20:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:20:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:20:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:20:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:20:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:20:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:20:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:20:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:20:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:20:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:20:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:20:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:20:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:20:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:20:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:20:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:20:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:20:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:20:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:20:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:20:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:20:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:20:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:20:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:20:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:20:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:20:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:20:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:20:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:20:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:20:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:20:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:20:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:20:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:20:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:20:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:20:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:20:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:20:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:20:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:20:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:20:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:20:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:20:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:21:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:21:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:21:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:21:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:21:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:21:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:21:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:21:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:21:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:21:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:21:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:21:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:21:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:21:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:21:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:21:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:21:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:21:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:21:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:21:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:21:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:21:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:21:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:21:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:21:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:21:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:21:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:21:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:21:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:21:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:21:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:21:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:21:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:21:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:21:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:21:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:21:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:21:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:21:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:21:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:21:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:21:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:21:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:21:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:21:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:21:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:21:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:21:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:21:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:21:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:21:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:21:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:21:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:21:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:21:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:21:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:21:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:21:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:21:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:21:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:21:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:21:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:21:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:21:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:21:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:21:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:21:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:21:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:21:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:21:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:21:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:21:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:21:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:22:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:22:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:22:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:22:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:22:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:22:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:22:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:22:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:22:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:22:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:22:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:22:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:22:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:22:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:22:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:22:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:22:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:22:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:22:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:22:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:22:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:22:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:22:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:22:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:22:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:22:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:22:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:22:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:22:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:22:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:22:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:22:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:22:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:22:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:22:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:22:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:22:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:22:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:22:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:22:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:22:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:22:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:22:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:22:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:22:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:22:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:22:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:22:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:22:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:22:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:22:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:22:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:22:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:22:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:22:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:22:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:22:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:22:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:22:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:22:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:22:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:22:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:22:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:22:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:22:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:22:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:22:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:22:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:22:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:23:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:23:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:23:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:23:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:23:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:23:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:23:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:23:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:23:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:23:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:23:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:23:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:23:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:23:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:23:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:23:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:23:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:23:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:23:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:23:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:23:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:23:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:23:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:23:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:23:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:23:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:23:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:23:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:23:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:23:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:23:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:23:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:23:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:23:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:23:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:23:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:23:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:23:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:23:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:23:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:23:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:23:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:23:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:23:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:23:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:23:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:23:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:23:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:23:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:23:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:23:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:23:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:23:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:23:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:23:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:23:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:23:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:23:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:23:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:23:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:23:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:23:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:23:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:23:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:23:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:23:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:23:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:23:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:23:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:23:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:23:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:23:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:23:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:24:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:24:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:24:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:24:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:24:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:24:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:24:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:24:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:24:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:24:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:24:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:24:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:24:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:24:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:24:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:24:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:24:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:24:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:24:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:24:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:24:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:24:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:24:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:24:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:24:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:24:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:24:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:24:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:24:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:24:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:24:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:24:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:24:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:24:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:24:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:24:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:24:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:24:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:24:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:24:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:24:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:24:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:24:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:24:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:24:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:24:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:24:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:24:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:24:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:24:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:24:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:24:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:24:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:24:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:24:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:24:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:24:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:24:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:24:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:24:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:24:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:24:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:24:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:24:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:24:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:24:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:24:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:24:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:24:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:24:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:24:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:24:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:25:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:25:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:25:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:25:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:25:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:25:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:25:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:25:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:25:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:25:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:25:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:25:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:25:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:25:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:25:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:25:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:25:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:25:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:25:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:25:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:25:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:25:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:25:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:25:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:25:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:25:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:25:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:25:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:25:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:25:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:25:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:25:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:25:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:25:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:25:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:25:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:25:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:25:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:25:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:25:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:25:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:25:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:25:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:25:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:25:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:25:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:25:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:25:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:25:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:25:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:25:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:25:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:25:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:25:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:25:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:25:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:25:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:25:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:25:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:25:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:25:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:25:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:25:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:25:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:26:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:26:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:26:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:26:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:26:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:26:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:26:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:26:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:26:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:26:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:26:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:26:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:26:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:26:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:26:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:26:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:26:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:26:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:26:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:26:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:26:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:26:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:26:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:26:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:26:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:26:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:26:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:26:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:26:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:26:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:26:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:26:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:26:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:26:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:26:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:26:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:26:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:26:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:26:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:26:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:26:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:26:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:26:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:26:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:26:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:26:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:26:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:26:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:26:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:26:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:26:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:26:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:26:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:26:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:26:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:26:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:26:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:26:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:26:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:26:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:26:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:26:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:26:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:26:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:26:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:26:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:26:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:26:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:26:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:27:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:27:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:27:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:27:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:27:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:27:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:27:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:27:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:27:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:27:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:27:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:27:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:27:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:27:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:27:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:27:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:27:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:27:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:27:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:27:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:27:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:27:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:27:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:27:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:27:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:27:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:27:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:27:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:27:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:27:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:27:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:27:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:27:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:27:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:27:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:27:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:27:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:27:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:27:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:27:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:27:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:27:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:27:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:27:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:27:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:27:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:27:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:27:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:27:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:27:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:27:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:27:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:27:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:27:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:27:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:27:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:27:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:27:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:27:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:27:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:27:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:27:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:27:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:27:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:27:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:27:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:27:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:27:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:27:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:27:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:27:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:27:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:27:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:27:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:28:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:28:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:28:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:28:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:28:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:28:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:28:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:28:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:28:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:28:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:28:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:28:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:28:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:28:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:28:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:28:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:28:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:28:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:28:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:28:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:28:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:28:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:28:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:28:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:28:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:28:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:28:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:28:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:28:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:28:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:28:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:28:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:28:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:28:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:28:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:28:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:28:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:28:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:28:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:28:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:28:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:28:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:28:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:28:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:28:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:28:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:28:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:28:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:28:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:28:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:28:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:28:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:28:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:28:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:28:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:28:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:28:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:28:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:28:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:28:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:28:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:28:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:28:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:28:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:28:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:28:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:28:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:28:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:28:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:28:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:28:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:28:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:28:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:29:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:29:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:29:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:29:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:29:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:29:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:29:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:29:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:29:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:29:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:29:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:29:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:29:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:29:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:29:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:29:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:29:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:29:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:29:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:29:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:29:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:29:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:29:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:29:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:29:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:29:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:29:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:29:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:29:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:29:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:29:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:29:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:29:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:29:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:29:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:29:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:29:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:29:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:29:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:29:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:29:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:29:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:29:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:29:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:29:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:29:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:29:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:29:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:29:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:29:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:29:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:29:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:29:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:29:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:29:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:29:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:29:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:29:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:29:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:29:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:29:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:29:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:29:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:29:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:29:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:29:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:29:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:29:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:30:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:30:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:30:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:30:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:30:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:30:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:30:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:30:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:30:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:30:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:30:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:30:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:30:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:30:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:30:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:30:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:30:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:30:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:30:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:30:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:30:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:30:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:30:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:30:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:30:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:30:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:30:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:30:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:30:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:30:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:30:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:30:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:30:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:30:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:30:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:30:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:30:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:30:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:30:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:30:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:30:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:30:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:30:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:30:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:30:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:30:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:30:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:30:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:30:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:30:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:30:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:30:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:30:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:30:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:30:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:30:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:30:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:30:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:30:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:30:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:30:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:30:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:30:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:30:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:30:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:30:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:30:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:30:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:30:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:31:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:31:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:31:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:31:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:31:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:31:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:31:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:31:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:31:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:31:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:31:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:31:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:31:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:31:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:31:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:31:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:31:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:31:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:31:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:31:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:31:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:31:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:31:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:31:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:31:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:31:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:31:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:31:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:31:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:31:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:31:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:31:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:31:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:31:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:31:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:31:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:31:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:31:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:31:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:31:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:31:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:31:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:31:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:31:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:31:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:31:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:31:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:31:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:31:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:31:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:31:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:31:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:31:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:31:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:31:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:31:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:31:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:31:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:31:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:31:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:31:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:31:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:31:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:31:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:31:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:31:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:31:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:31:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:31:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:31:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:31:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:31:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:31:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:32:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:32:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:32:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:32:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:32:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:32:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:32:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:32:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:32:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:32:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:32:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:32:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:32:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:32:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:32:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:32:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:32:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:32:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:32:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:32:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:32:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:32:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:32:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:32:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:32:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:32:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:32:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:32:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:32:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:32:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:32:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:32:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:32:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:32:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:32:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:32:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:32:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:32:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:32:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:32:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:32:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:32:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:32:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:32:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:32:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:32:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:32:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:32:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:32:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:32:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:32:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:32:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:32:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:32:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:32:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:32:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:32:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:32:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:32:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:32:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:32:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:32:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:32:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:32:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:32:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:32:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:32:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:32:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:33:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:33:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:33:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:33:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:33:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:33:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:33:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:33:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:33:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:33:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:33:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:33:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:33:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:33:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:33:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:33:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:33:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:33:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:33:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:33:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:33:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:33:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:33:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:33:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:33:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:33:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:33:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:33:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:33:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:33:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:33:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:33:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:33:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:33:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:33:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:33:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:33:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:33:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:33:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:33:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:33:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:33:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:33:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:33:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:33:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:33:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:33:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:33:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:33:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:33:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:33:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:33:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:33:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:33:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:33:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:33:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:33:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:33:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:33:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:33:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:33:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:33:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:33:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:33:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:33:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:33:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:33:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:33:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:33:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:33:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:33:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:33:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:33:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:33:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:34:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:34:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:34:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:34:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:34:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:34:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:34:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:34:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:34:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:34:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:34:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:34:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:34:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:34:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:34:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:34:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:34:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:34:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:34:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:34:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:34:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:34:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:34:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:34:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:34:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:34:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:34:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:34:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:34:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:34:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:34:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:34:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:34:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:34:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:34:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:34:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:34:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:34:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:34:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:34:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:34:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:34:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:34:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:34:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:34:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:34:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:34:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:34:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:34:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:34:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:34:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:34:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:34:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:34:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:34:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:34:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:34:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:34:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:34:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:34:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:34:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:34:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:34:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:34:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:34:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:34:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:34:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:34:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:34:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:35:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:35:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:35:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:35:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:35:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:35:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:35:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:35:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:35:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:35:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:35:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:35:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:35:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:35:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:35:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:35:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:35:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:35:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:35:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:35:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:35:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:35:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:35:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:35:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:35:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:35:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:35:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:35:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:35:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:35:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:35:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:35:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:35:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:35:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:35:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:35:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:35:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:35:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:35:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:35:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:35:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:35:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:35:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:35:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:35:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:35:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:35:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:35:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:35:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:35:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:35:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:35:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:35:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:35:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:35:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:35:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:35:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:35:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:35:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:35:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:35:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:35:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:35:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:35:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:35:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:35:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:35:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:35:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:35:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:35:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:35:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:35:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:35:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:36:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:36:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:36:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:36:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:36:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:36:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:36:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:36:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:36:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:36:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:36:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:36:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:36:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:36:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:36:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:36:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:36:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:36:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:36:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:36:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:36:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:36:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:36:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:36:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:36:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:36:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:36:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:36:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:36:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:36:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:36:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:36:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:36:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:36:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:36:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:36:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:36:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:36:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:36:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:36:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:36:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:36:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:36:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:36:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:36:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:36:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:36:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:36:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:36:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:36:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:36:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:36:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:36:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:36:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:36:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:36:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:36:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:36:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:36:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:36:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:36:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:36:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:36:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:36:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:36:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:36:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:36:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:36:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:36:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:36:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:36:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:36:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:36:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:36:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:37:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:37:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:37:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:37:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:37:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:37:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:37:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:37:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:37:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:37:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:37:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:37:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:37:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:37:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:37:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:37:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:37:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:37:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:37:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:37:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:37:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:37:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:37:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:37:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:37:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:37:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:37:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:37:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:37:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:37:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:37:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:37:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:37:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:37:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-15 18:37:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-15 18:37:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-15 18:37:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-15 18:37:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-15 18:37:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-15 18:37:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-15 18:37:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-15 18:37:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-15 18:37:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-15 18:37:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-15 18:37:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-15 18:37:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-15 18:37:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-15 18:37:37 WRN Prometheus returned 401. Re-authenticating... 2025-09-15 18:37:37 INF Re-authenticating Prometheus client 2025-09-15 18:37:37 WRN Prometheus returned 401. Re-authenticating... 2025-09-15 18:37:37 WRN Prometheus returned 401. Re-authenticating... 2025-09-15 18:37:37 WRN Prometheus returned 401. Re-authenticating... 2025-09-15 18:37:37 WRN Prometheus returned 401. Re-authenticating... 2025-09-15 18:37:37 WRN Prometheus returned 401. Re-authenticating... 2025-09-15 23:29:34 INF Adding event: Scenario_benchmark-stop 2025-09-15 23:52:28 INF Stopping Lodemon 2025-09-15 23:52:28 INF Stopping monitoring service...