==================================================================================================== ============================ Pod describe lodemon-74f4bf6c95-xwgl2 ============================ ==================================================================================================== Name: lodemon-74f4bf6c95-xwgl2 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Fri, 19 Sep 2025 15:34:53 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=74f4bf6c95 skaffold.dev/run-id=7f99752e-6b72-4da0-a3dc-9e7b7204e346 Annotations: Status: Running IP: 10.106.40.163 IPs: IP: 10.106.40.163 Controlled By: ReplicaSet/lodemon-74f4bf6c95 Containers: lodemon: Container ID: containerd://f39c8998be48ed820c617f5e8bbb92b0a9f436a719db1360fdabf5889114a835 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Fri, 19 Sep 2025 15:34:54 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p5swr (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-p5swr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-74f4bf6c95-xwgl2 ============================== ==================================================================================================== 2025-09-19 15:34:54 INF Starting lodemon server on port 8080 2025-09-19 15:34:54 INF Initializing clients... 2025-09-19 15:34:54 DBG Fetching ID token for accessing prometheus 2025-09-19 15:34:54 DBG Fetched ID token successfully 2025-09-19 15:34:54 INF Clients initialized successfully 2025-09-19 15:35:29 INF Configuring Lodemon 2025-09-19 15:35:29 INF Configuring monitoring units 2025-09-19 15:35:29 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-19 15:35:29 TRC Parsing Prometheus queries from config 2025-09-19 15:35:29 TRC Parsing component query: DS Java heap mem usage 2025-09-19 15:35:29 TRC Parsing component query: IDM Java heap mem usage 2025-09-19 15:35:29 TRC Parsing component query: AM Java heap mem usage 2025-09-19 15:35:29 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-19 15:35:29 TRC Parsing global query: 2025-09-19 15:35:29 TRC Parsing scoped query: DS Identity backend connections 2025-09-19 15:35:29 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-19 15:35:29 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-19 15:35:29 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-19 15:35:29 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-19 15:35:29 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-19 15:35:29 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-19 15:35:29 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-19 15:35:29 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-19 15:35:29 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-19 15:35:29 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-19 15:35:29 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-19 15:35:29 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-19 15:35:29 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-19 15:35:29 INF  - All monitoring units configured 2025-09-19 15:35:29 INF Configuring monitoring service 2025-09-19 15:35:29 INF  - Adding monitoring units to the service 2025-09-19 15:35:29 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-19 15:35:29 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-19 15:35:29 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-19 15:35:29 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-19 15:35:29 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-19 15:35:29 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-19 15:35:29 INF  - All monitoring units added to the service 2025-09-19 15:35:29 INF Starting Lodemon now 2025-09-19 15:35:29 INF Starting monitoring service... 2025-09-19 15:35:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:35:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:35:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:35:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:35:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:35:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:35:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:35:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:35:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:35:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:35:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:35:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:35:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:31 INF Adding event: Scenario_benchmark 2025-09-19 15:35:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:35:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:35:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:35:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:35:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:35:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:35:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:35:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:35:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:35:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:35:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:35:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:35:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:35:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:35:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:35:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:35:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:35:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:35:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:35:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:35:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:35:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:35:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:35:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:35:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:35:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:35:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:36:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:36:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:36:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:36:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:36:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:36:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:36:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:36:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:36:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:36:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:36:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:36:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:36:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:36:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:36:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:36:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:36:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:36:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:36:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:36:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:36:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:36:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:36:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:36:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:36:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:36:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:36:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:36:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:36:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:36:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:36:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:36:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:36:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:36:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:36:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:36:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:36:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:36:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:36:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:36:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:36:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:36:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:36:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:36:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:36:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:36:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:36:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:36:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:36:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:36:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:36:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:36:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:36:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:36:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:36:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:36:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:36:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:36:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:36:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:36:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:36:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:36:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:36:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:36:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:36:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:36:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:36:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:36:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:36:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:36:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:36:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:36:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:36:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:36:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:37:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:37:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:37:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:37:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:37:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:37:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:37:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:37:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:37:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:37:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:37:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:37:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:37:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:37:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:37:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:37:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:37:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:37:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:37:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:37:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:37:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:37:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:37:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:37:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:37:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:37:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:37:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:37:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:37:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:37:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:37:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:37:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:37:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:37:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:37:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:37:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:37:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:37:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:37:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:37:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:37:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:37:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:37:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:37:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:37:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:37:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:37:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:37:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:37:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:37:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:37:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:37:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:37:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:37:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:37:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:37:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:37:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:37:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:37:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:37:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:37:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:37:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:37:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:37:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:37:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:37:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:37:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:37:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:37:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:37:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:37:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:37:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:37:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:38:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:38:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:38:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:38:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:38:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:38:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:38:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:38:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:38:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:38:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:38:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:38:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:38:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:38:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:38:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:38:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:38:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:38:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:38:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:38:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:38:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:38:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:38:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:38:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:38:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:38:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:38:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:38:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:38:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:38:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:38:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:38:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:38:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:38:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:38:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:38:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:38:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:38:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:38:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:38:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:38:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:38:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:38:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:38:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:38:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:38:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:38:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:38:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:38:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:38:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:38:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:38:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:38:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:38:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:38:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:38:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:38:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:38:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:38:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:38:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:38:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:38:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:38:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:38:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:38:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:38:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:38:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:38:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:39:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:39:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:39:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:39:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:39:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:39:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:39:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:39:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:39:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:39:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:39:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:39:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:39:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:39:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:39:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:39:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:39:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:39:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:39:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:39:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:39:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:39:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:39:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:39:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:39:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:39:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:39:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:39:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:39:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:39:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:39:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:39:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:39:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:39:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:39:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:39:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:39:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:39:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:39:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:39:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:39:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:39:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:39:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:39:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:39:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:39:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:39:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:39:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:39:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:39:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:39:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:39:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:39:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:39:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:39:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:39:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:39:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:39:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:39:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:39:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:39:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:39:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:39:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:39:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:39:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:39:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:39:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:39:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:40:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:40:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:40:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:40:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:40:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:40:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:40:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:40:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:40:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:40:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:40:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:40:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:40:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:40:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:40:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:40:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:40:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:40:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:40:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:40:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:40:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:40:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:40:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:40:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:40:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:40:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:40:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:40:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:40:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:40:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:40:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:40:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:40:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:40:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:40:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:40:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:40:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:40:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:40:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:40:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:40:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:40:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:40:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:40:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:40:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:40:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:40:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:40:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:40:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:40:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:40:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:40:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:40:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:40:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:40:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:40:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:40:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:40:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:40:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:40:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:40:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:40:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:40:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:40:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:40:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:40:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:40:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:40:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:40:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:40:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:40:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:40:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:40:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:40:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:41:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:41:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:41:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:41:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:41:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:41:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:41:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:41:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:41:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:41:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:41:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:41:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:41:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:41:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:41:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:41:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:41:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:41:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:41:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:41:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:41:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:41:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:41:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:41:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:41:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:41:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:41:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:41:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:41:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:41:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:41:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:41:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:41:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:41:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:41:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:41:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:41:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:41:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:41:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:41:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:41:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:41:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:41:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:41:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:41:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:41:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:41:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:41:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:41:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:41:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:41:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:41:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:41:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:41:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:41:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:41:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:41:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:41:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:41:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:41:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:41:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:41:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:41:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:41:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:41:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:41:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:41:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:41:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:41:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:41:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:41:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:41:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:41:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:41:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:42:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:42:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:42:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:42:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:42:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:42:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:42:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:42:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:42:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:42:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:42:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:42:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:42:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:42:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:42:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:42:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:42:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:42:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:42:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:42:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:42:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:42:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:42:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:42:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:42:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:42:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:42:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:42:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:42:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:42:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:42:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:42:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:42:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:42:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:42:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:42:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:42:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:42:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:42:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:42:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:42:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:42:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:42:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:42:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:42:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:42:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:42:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:42:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:42:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:42:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:42:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:42:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:42:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:42:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:42:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:42:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:42:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:42:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:42:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:42:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:42:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:42:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:42:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:42:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:42:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:42:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:42:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:42:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:43:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:43:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:43:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:43:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:43:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:43:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:43:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:43:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:43:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:43:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:43:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:43:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:43:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:43:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:43:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:43:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:43:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:43:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:43:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:43:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:43:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:43:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:43:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:43:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:43:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:43:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:43:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:43:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:43:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:43:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:43:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:43:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:43:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:43:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:43:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:43:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:43:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:43:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:43:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:43:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:43:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:43:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:43:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:43:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:43:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:43:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:43:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:43:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:43:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:43:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:43:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:43:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:43:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:43:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:43:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:43:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:43:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:43:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:43:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:43:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:43:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:43:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:43:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:43:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:43:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:43:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:43:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:43:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:44:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:44:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:44:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:44:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:44:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:44:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:44:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:44:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:44:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:44:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:44:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:44:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:44:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:44:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:44:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:44:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:44:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:44:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:44:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:44:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:44:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:44:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:44:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:44:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:44:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:44:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:44:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:44:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:44:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:44:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:44:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:44:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:44:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:44:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:44:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:44:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:44:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:44:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:44:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:44:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:44:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:44:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:44:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:44:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:44:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:44:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:44:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:44:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:44:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:44:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:44:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:44:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:44:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:44:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:44:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:44:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:44:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:44:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:44:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:44:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:44:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:44:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:44:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:44:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:44:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:44:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:44:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:44:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:44:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:45:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:45:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:45:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:45:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:45:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:45:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:45:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:45:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:45:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:45:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:45:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:45:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:45:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:45:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:45:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:45:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:45:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:45:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:45:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:45:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:45:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:45:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:45:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:45:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:45:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:45:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:45:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:45:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:45:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:45:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:45:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:45:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:45:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:45:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:45:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:45:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:45:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:45:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:45:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:45:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:45:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:45:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:45:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:45:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:45:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:45:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:45:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:45:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:45:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:45:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:45:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:45:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:45:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:45:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:45:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:45:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:45:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:45:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:45:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:45:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:45:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:45:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:45:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:45:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:45:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:45:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:45:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:45:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:45:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:45:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:45:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:45:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:45:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:46:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:46:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:46:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:46:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:46:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:46:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:46:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:46:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:46:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:46:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:46:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:46:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:46:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:46:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:46:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:46:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:46:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:46:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:46:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:46:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:46:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:46:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:46:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:46:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:46:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:46:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:46:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:46:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:46:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:46:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:46:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:46:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:46:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:46:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:46:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:46:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:46:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:46:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:46:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:46:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:46:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:46:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:46:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:46:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:46:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:46:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:46:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:46:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:46:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:46:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:46:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:46:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:46:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:46:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:46:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:46:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:46:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:46:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:46:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:46:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:46:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:46:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:46:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:46:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:46:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:46:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:46:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:46:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:46:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:47:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:47:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:47:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:47:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:47:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:47:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:47:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:47:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:47:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:47:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:47:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:47:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:47:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:47:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:47:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:47:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:47:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:47:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:47:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:47:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:47:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:47:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:47:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:47:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:47:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:47:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:47:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:47:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:47:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:47:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:47:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:47:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:47:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:47:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:47:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:47:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:47:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:47:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:47:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:47:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:47:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:47:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:47:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:47:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:47:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:47:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:47:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:47:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:47:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:47:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:47:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:47:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:47:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:47:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:47:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:47:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:47:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:47:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:47:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:47:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:47:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:47:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:47:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:47:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:47:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:47:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:47:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:47:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:47:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:47:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:47:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:47:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:47:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:48:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:48:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:48:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:48:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:48:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:48:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:48:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:48:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:48:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:48:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:48:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:48:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:48:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:48:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:48:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:48:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:48:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:48:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:48:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:48:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:48:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:48:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:48:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:48:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:48:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:48:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:48:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:48:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:48:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:48:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:48:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:48:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:48:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:48:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:48:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:48:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:48:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:48:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:48:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:48:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:48:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:48:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:48:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:48:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:48:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:48:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:48:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:48:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:48:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:48:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:48:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:48:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:48:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:48:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:48:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:48:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:48:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:48:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:48:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:48:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:48:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:48:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:48:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:48:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:48:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:48:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:48:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:48:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:48:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:49:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:49:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:49:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:49:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:49:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:49:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:49:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:49:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:49:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:49:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:49:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:49:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:49:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:49:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:49:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:49:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:49:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:49:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:49:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:49:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:49:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:49:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:49:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:49:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:49:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:49:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:49:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:49:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:49:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:49:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:49:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:49:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:49:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:49:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:49:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:49:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:49:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:49:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:49:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:49:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:49:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:49:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:49:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:49:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:49:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:49:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:49:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:49:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:49:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:49:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:49:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:49:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:49:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:49:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:49:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:49:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:49:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:49:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:49:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:49:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:49:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:49:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:49:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:49:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:49:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:49:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:49:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:49:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:49:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:49:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:49:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:49:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:49:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:49:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:50:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:50:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:50:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:50:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:50:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:50:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:50:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:50:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:50:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:50:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:50:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:50:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:50:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:50:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:50:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:50:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:50:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:50:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:50:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:50:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:50:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:50:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:50:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:50:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:50:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:50:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:50:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:50:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:50:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:50:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:50:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:50:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:50:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:50:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:50:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:50:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:50:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:50:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:50:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:50:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:50:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:50:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:50:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:50:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:50:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:50:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:50:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:50:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:50:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:50:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:50:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:50:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:50:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:50:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:50:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:50:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:50:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:50:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:50:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:50:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:50:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:50:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:50:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:50:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:50:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:50:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:50:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:50:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:50:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:50:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:50:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:50:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:50:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:51:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:51:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:51:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:51:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:51:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:51:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:51:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:51:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:51:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:51:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:51:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:51:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:51:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:51:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:51:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:51:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:51:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:51:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:51:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:51:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:51:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:51:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:51:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:51:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:51:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:51:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:51:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:51:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:51:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:51:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:51:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:51:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:51:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:51:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:51:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:51:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:51:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:51:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:51:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:51:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:51:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:51:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:51:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:51:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:51:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:51:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:51:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:51:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:51:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:51:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:51:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:51:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:51:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:51:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:51:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:51:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:51:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:51:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:51:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:51:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:51:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:51:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:51:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:51:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:51:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:51:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:51:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:51:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:51:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:52:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:52:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:52:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:52:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:52:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:52:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:52:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:52:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:52:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:52:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:52:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:52:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:52:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:52:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:52:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:52:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:52:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:52:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:52:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:52:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:52:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:52:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:52:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:52:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:52:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:52:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:52:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:52:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:52:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:52:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:52:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:52:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:52:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:52:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:52:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:52:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:52:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:52:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:52:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:52:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:52:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:52:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:52:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:52:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:52:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:52:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:52:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:52:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:52:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:52:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:52:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:52:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:52:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:52:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:52:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:52:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:52:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:52:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:52:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:52:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:52:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:52:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:53:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:53:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:53:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:53:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:53:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:53:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:53:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:53:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:53:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:53:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:53:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:53:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:53:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:53:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:53:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:53:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:53:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:53:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:53:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:53:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:53:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:53:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:53:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:53:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:53:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:53:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:53:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:53:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:53:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:53:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:53:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:53:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:53:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:53:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:53:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:53:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:53:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:53:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:53:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:53:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:53:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:53:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:53:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:53:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:53:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:53:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:53:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:53:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:53:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:53:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:53:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:53:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:53:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:53:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:53:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:53:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:53:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:53:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:53:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:53:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:53:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:53:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:53:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:53:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:53:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:53:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:53:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:53:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:53:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:53:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:53:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:53:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:53:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:53:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:54:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:54:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:54:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:54:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:54:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:54:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:54:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:54:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:54:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:54:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:54:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:54:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:54:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:54:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:54:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:54:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:54:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:54:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:54:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:54:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:54:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:54:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:54:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:54:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:54:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:54:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:54:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:54:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:54:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:54:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:54:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:54:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:54:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:54:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:54:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:54:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:54:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:54:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:54:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:54:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:54:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:54:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:54:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:54:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:54:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:54:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:54:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:54:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:54:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:54:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:54:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:54:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:54:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:54:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:54:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:54:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:54:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:54:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:54:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:54:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:54:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:54:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:54:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:54:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:54:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:54:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:54:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:54:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:54:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:54:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:54:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:54:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:54:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:55:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:55:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:55:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:55:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:55:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:55:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:55:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:55:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:55:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:55:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:55:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:55:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:55:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:55:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:55:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:55:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:55:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:55:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:55:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:55:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:55:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:55:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:55:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:55:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:55:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:55:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:55:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:55:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:55:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:55:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:55:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:55:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:55:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:55:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:55:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:55:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:55:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:55:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:55:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:55:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:55:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:55:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:55:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:55:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:55:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:55:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:55:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:55:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:55:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:55:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:55:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:55:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:55:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:55:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:55:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:55:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:55:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:55:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:55:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:55:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:55:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:55:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:55:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:55:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:55:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:55:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:55:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:55:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:56:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:56:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:56:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:56:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:56:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:56:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:56:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:56:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:56:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:56:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:56:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:56:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:56:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:56:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:56:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:56:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:56:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:56:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:56:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:56:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:56:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:56:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:56:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:56:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:56:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:56:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:56:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:56:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:56:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:56:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:56:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:56:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:56:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:56:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:56:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:56:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:56:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:56:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:56:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:56:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:56:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:56:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:56:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:56:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:56:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:56:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:56:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:56:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:56:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:56:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:56:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:56:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:56:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:56:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:56:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:56:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:56:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:56:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:56:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:56:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:56:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:56:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:56:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:56:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:56:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:56:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:56:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:56:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:57:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:57:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:57:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:57:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:57:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:57:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:57:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:57:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:57:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:57:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:57:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:57:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:57:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:57:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:57:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:57:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:57:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:57:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:57:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:57:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:57:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:57:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:57:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:57:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:57:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:57:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:57:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:57:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:57:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:57:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:57:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:57:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:57:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:57:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:57:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:57:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:57:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:57:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:57:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:57:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:57:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:57:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:57:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:57:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:57:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:57:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:57:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:57:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:57:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:57:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:57:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:57:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:57:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:57:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:57:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:57:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:57:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:57:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:57:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:57:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:57:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:57:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:57:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:57:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:57:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:57:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:57:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:57:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:57:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:57:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:57:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:57:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:57:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:57:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:58:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:58:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:58:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:58:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:58:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:58:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:58:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:58:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:58:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:58:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:58:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:58:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:58:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:58:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:58:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:58:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:58:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:58:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:58:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:58:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:58:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:58:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:58:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:58:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:58:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:58:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:58:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:58:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:58:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:58:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:58:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:58:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:58:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:58:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:58:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:58:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:58:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:58:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:58:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:58:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:58:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:58:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:58:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:58:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:58:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:58:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:58:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:58:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:58:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:58:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:58:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:58:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:58:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:58:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:58:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:58:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:58:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:58:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:58:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:58:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:58:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:58:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:58:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:58:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:58:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:58:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:58:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:58:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:58:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:58:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:58:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:58:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:58:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:58:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:59:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:59:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:59:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:59:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:59:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:59:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:59:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:59:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:59:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:59:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:59:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:59:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:59:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:59:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:59:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:59:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:59:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:59:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:59:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:59:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:59:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:59:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:59:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:59:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:59:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:59:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:59:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:59:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:59:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:59:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:59:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:59:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:59:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:59:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:59:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:59:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:59:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:59:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:59:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:59:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:59:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:59:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:59:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:59:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:59:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:59:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:59:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:59:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:59:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:59:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:59:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:59:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:59:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:59:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:59:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 15:59:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 15:59:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 15:59:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 15:59:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 15:59:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 15:59:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 15:59:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 15:59:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 15:59:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 15:59:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 15:59:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 15:59:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 15:59:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 15:59:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:00:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:00:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:00:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:00:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:00:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:00:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:00:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:00:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:00:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:00:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:00:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:00:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:00:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:00:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:00:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:00:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:00:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:00:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:00:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:00:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:00:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:00:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:00:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:00:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:00:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:00:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:00:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:00:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:00:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:00:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:00:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:00:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:00:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:00:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:00:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:00:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:00:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:00:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:00:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:00:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:00:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:00:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:00:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:00:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:00:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:00:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:00:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:00:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:00:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:00:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:00:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:00:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:00:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:00:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:00:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:00:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:00:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:00:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:00:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:00:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:00:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:00:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:00:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:00:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:00:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:00:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:00:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:00:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:00:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:00:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:00:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:00:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:00:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:01:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:01:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:01:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:01:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:01:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:01:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:01:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:01:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:01:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:01:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:01:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:01:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:01:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:01:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:01:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:01:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:01:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:01:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:01:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:01:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:01:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:01:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:01:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:01:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:01:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:01:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:01:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:01:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:01:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:01:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:01:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:01:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:01:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:01:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:01:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:01:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:01:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:01:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:01:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:01:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:01:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:01:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:01:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:01:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:01:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:01:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:01:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:01:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:01:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:01:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:01:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:01:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:01:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:01:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:01:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:01:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:01:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:01:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:01:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:01:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:01:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:01:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:01:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:01:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:01:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:01:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:01:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:01:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:02:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:02:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:02:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:02:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:02:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:02:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:02:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:02:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:02:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:02:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:02:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:02:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:02:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:02:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:02:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:02:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:02:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:02:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:02:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:02:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:02:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:02:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:02:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:02:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:02:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:02:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:02:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:02:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:02:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:02:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:02:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:02:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:02:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:02:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:02:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:02:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:02:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:02:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:02:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:02:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:02:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:02:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:02:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:02:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:02:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:02:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:02:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:02:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:02:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:02:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:02:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:02:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:02:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:02:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:02:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:02:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:02:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:02:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:02:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:02:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:02:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:02:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:02:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:02:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:02:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:02:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:02:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:02:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:03:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:03:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:03:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:03:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:03:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:03:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:03:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:03:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:03:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:03:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:03:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:03:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:03:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:03:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:03:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:03:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:03:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:03:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:03:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:03:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:03:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:03:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:03:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:03:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:03:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:03:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:03:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:03:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:03:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:03:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:03:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:03:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:03:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:03:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:03:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:03:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:03:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:03:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:03:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:03:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:03:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:03:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:03:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:03:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:03:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:03:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:03:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:03:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:03:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:03:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:03:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:03:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:03:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:03:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:03:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:03:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:03:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:03:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:03:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:03:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:03:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:03:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:03:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:03:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:03:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:03:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:03:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:03:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:04:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:04:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:04:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:04:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:04:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:04:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:04:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:04:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:04:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:04:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:04:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:04:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:04:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:04:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:04:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:04:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:04:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:04:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:04:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:04:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:04:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:04:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:04:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:04:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:04:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:04:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:04:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:04:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:04:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:04:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:04:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:04:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:04:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:04:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:04:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:04:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:04:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:04:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:04:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:04:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:04:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:04:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:04:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:04:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:04:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:04:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:04:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:04:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:04:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:04:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:04:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:04:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:04:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:04:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:04:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:04:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:04:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:04:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:04:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:04:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:04:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:04:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:04:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:04:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:04:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:04:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:04:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:04:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:04:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:05:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:05:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:05:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:05:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:05:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:05:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:05:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:05:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:05:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:05:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:05:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:05:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:05:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:05:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:05:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:05:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:05:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:05:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:05:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:05:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:05:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:05:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:05:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:05:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:05:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:05:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:05:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:05:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:05:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:05:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:05:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:05:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:05:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:05:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:05:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:05:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:05:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:05:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:05:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:05:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:05:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:05:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:05:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:05:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:05:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:05:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:05:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:05:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:05:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:05:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:05:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:05:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:05:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:05:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:05:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:05:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:05:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:05:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:05:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:05:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:05:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:05:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:05:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:05:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:05:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:05:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:05:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:05:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:05:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:05:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:05:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:05:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:05:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:05:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:06:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:06:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:06:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:06:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:06:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:06:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:06:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:06:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:06:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:06:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:06:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:06:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:06:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:06:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:06:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:06:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:06:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:06:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:06:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:06:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:06:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:06:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:06:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:06:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:06:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:06:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:06:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:06:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:06:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:06:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:06:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:06:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:06:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:06:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:06:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:06:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:06:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:06:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:06:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:06:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:06:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:06:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:06:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:06:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:06:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:06:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:06:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:06:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:06:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:06:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:06:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:06:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:06:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:06:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:06:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:06:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:06:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:06:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:06:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:06:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:06:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:06:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:06:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:06:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:06:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:06:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:06:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:06:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:06:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:06:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:06:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:06:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:06:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:07:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:07:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:07:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:07:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:07:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:07:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:07:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:07:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:07:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:07:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:07:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:07:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:07:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:07:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:07:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:07:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:07:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:07:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:07:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:07:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:07:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:07:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:07:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:07:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:07:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:07:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:07:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:07:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:07:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:07:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:07:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:07:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:07:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:07:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:07:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:07:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:07:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:07:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:07:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:07:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:07:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:07:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:07:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:07:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:07:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:07:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:07:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:07:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:07:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:07:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:07:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:07:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:07:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:07:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:07:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:07:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:07:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:07:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:07:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:07:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:07:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:07:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:07:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:07:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:07:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:07:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:07:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:07:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:07:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:08:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:08:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:08:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:08:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:08:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:08:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:08:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:08:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:08:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:08:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:08:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:08:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:08:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:08:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:08:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:08:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:08:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:08:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:08:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:08:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:08:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:08:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:08:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:08:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:08:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:08:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:08:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:08:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:08:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:08:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:08:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:08:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:08:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:08:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:08:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:08:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:08:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:08:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:08:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:08:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:08:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:08:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:08:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:08:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:08:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:08:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:08:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:08:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:08:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:08:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:08:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:08:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:08:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:08:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:08:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:08:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:08:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:08:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:08:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:08:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:08:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:08:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:09:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:09:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:09:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:09:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:09:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:09:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:09:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:09:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:09:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:09:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:09:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:09:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:09:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:09:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:09:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:09:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:09:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:09:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:09:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:09:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:09:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:09:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:09:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:09:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:09:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:09:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:09:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:09:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:09:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:09:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:09:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:09:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:09:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:09:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:09:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:09:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:09:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:09:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:09:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:09:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:09:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:09:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:09:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:09:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:09:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:09:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:09:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:09:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:09:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:09:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:09:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:09:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:09:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:09:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:09:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:09:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:09:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:09:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:09:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:09:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:09:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:09:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:09:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:09:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:09:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:09:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:09:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:09:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:09:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:09:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:09:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:09:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:09:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:10:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:10:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:10:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:10:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:10:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:10:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:10:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:10:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:10:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:10:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:10:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:10:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:10:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:10:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:10:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:10:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:10:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:10:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:10:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:10:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:10:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:10:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:10:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:10:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:10:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:10:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:10:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:10:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:10:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:10:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:10:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:10:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:10:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:10:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:10:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:10:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:10:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:10:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:10:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:10:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:10:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:10:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:10:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:10:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:10:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:10:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:10:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:10:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:10:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:10:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:10:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:10:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:10:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:10:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:10:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:10:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:10:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:10:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:10:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:10:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:10:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:10:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:10:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:10:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:10:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:10:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:10:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:10:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:11:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:11:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:11:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:11:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:11:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:11:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:11:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:11:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:11:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:11:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:11:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:11:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:11:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:11:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:11:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:11:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:11:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:11:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:11:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:11:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:11:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:11:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:11:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:11:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:11:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:11:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:11:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:11:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:11:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:11:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:11:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:11:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:11:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:11:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:11:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:11:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:11:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:11:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:11:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:11:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:11:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:11:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:11:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:11:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:11:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:11:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:11:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:11:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:11:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:11:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:11:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:11:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:11:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:11:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:11:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:11:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:11:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:11:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:11:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:11:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:11:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:11:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:11:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:11:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:11:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:11:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:11:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:11:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:11:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:12:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:12:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:12:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:12:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:12:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:12:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:12:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:12:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:12:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:12:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:12:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:12:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:12:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:12:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:12:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:12:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:12:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:12:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:12:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:12:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:12:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:12:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:12:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:12:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:12:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:12:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:12:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:12:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:12:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:12:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:12:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:12:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:12:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:12:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:12:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:12:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:12:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:12:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:12:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:12:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:12:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:12:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:12:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:12:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:12:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:12:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:12:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:12:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:12:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:12:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:12:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:12:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:12:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:12:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:12:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:12:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:12:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:12:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:12:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:12:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:12:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:12:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:12:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:12:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:12:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:12:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:12:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:12:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:12:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:12:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:12:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:12:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:12:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:12:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:13:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:13:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:13:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:13:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:13:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:13:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:13:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:13:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:13:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:13:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:13:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:13:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:13:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:13:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:13:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:13:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:13:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:13:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:13:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:13:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:13:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:13:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:13:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:13:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:13:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:13:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:13:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:13:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:13:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:13:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:13:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:13:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:13:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:13:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:13:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:13:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:13:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:13:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:13:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:13:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:13:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:13:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:13:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:13:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:13:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:13:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:13:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:13:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:13:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:13:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:13:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:13:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:13:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:13:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:13:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:13:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:13:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:13:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:13:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:13:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:13:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:13:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:13:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:13:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:13:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:13:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:13:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:13:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:13:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:13:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:13:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:13:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:13:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:13:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:14:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:14:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:14:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:14:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:14:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:14:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:14:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:14:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:14:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:14:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:14:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:14:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:14:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:14:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:14:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:14:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:14:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:14:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:14:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:14:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:14:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:14:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:14:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:14:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:14:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:14:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:14:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:14:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:14:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:14:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:14:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:14:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:14:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:14:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:14:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:14:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:14:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:14:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:14:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:14:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:14:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:14:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:14:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:14:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:14:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:14:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:14:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:14:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:14:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:14:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:14:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:14:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:14:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:14:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:14:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:14:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:14:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:14:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:14:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:14:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:14:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:14:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:14:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:14:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:14:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:14:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:14:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:14:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:14:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:14:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:14:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:14:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:14:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:14:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:15:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:15:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:15:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:15:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:15:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:15:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:15:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:15:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:15:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:15:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:15:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:15:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:15:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:15:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:15:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:15:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:15:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:15:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:15:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:15:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:15:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:15:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:15:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:15:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:15:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:15:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:15:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:15:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:15:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:15:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:15:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:15:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:15:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:15:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:15:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:15:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:15:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:15:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:15:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:15:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:15:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:15:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:15:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:15:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:15:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:15:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:15:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:15:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:15:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:15:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:15:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:15:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:15:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:15:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:15:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:15:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:15:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:15:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:15:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:15:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:15:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:15:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:15:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:15:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:15:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:15:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:15:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:15:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:15:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:15:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:15:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:15:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:15:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:15:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:16:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:16:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:16:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:16:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:16:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:16:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:16:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:16:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:16:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:16:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:16:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:16:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:16:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:16:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:16:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:16:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:16:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:16:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:16:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:16:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:16:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:16:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:16:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:16:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:16:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:16:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:16:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:16:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:16:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:16:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:16:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:16:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:16:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:16:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:16:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:16:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:16:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:16:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:16:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:16:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:16:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:16:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:16:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:16:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:16:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:16:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:16:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:16:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:16:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:16:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:16:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:16:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:16:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:16:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:16:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:16:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:16:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:16:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:16:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:16:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:16:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:16:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:16:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:16:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:16:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:16:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:16:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:16:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:16:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:16:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:16:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:16:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:16:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:17:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:17:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:17:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:17:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:17:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:17:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:17:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:17:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:17:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:17:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:17:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:17:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:17:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:17:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:17:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:17:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:17:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:17:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:17:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:17:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:17:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:17:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:17:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:17:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:17:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:17:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:17:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:17:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:17:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:17:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:17:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:17:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:17:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:17:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:17:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:17:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:17:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:17:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:17:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:17:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:17:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:17:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:17:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:17:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:17:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:17:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:17:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:17:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:17:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:17:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:17:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:17:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:17:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:17:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:17:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:17:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:17:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:17:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:17:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:17:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:17:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:17:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:17:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:17:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:17:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:17:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:17:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:17:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:18:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:18:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:18:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:18:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:18:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:18:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:18:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:18:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:18:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:18:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:18:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:18:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:18:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:18:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:18:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:18:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:18:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:18:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:18:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:18:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:18:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:18:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:18:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:18:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:18:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:18:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:18:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:18:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:18:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:18:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:18:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:18:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:18:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:18:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:18:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:18:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:18:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:18:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:18:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:18:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:18:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:18:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:18:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:18:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:18:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:18:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:18:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:18:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:18:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:18:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:18:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:18:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:18:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:18:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:18:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:18:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:18:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:18:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:18:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:18:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:18:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:18:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:18:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:18:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:18:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:18:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:18:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:18:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:18:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:19:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:19:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:19:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:19:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:19:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:19:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:19:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:19:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:19:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:19:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:19:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:19:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:19:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:19:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:19:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:19:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:19:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:19:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:19:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:19:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:19:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:19:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:19:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:19:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:19:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:19:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:19:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:19:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:19:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:19:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:19:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:19:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:19:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:19:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:19:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:19:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:19:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:19:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:19:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:19:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:19:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:19:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:19:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:19:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:19:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:19:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:19:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:19:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:19:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:19:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:19:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:19:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:19:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:19:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:19:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:19:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:19:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:19:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:19:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:19:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:19:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:19:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:19:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:19:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:19:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:19:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:19:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:19:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:19:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:19:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:19:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:19:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:19:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:19:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:20:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:20:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:20:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:20:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:20:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:20:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:20:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:20:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:20:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:20:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:20:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:20:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:20:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:20:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:20:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:20:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:20:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:20:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:20:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:20:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:20:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:20:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:20:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:20:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:20:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:20:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:20:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:20:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:20:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:20:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:20:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:20:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:20:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:20:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:20:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:20:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:20:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:20:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:20:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:20:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:20:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:20:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:20:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:20:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:20:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:20:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:20:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:20:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:20:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:20:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:20:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:20:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:20:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:20:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:20:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:20:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:20:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:20:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:20:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:20:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:20:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:20:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:20:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:20:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:20:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:20:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:20:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:20:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:20:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:20:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:20:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:20:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:20:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:20:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:21:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:21:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:21:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:21:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:21:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:21:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:21:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:21:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:21:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:21:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:21:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:21:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:21:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:21:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:21:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:21:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:21:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:21:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:21:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:21:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:21:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:21:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:21:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:21:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:21:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:21:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:21:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:21:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:21:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:21:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:21:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:21:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:21:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:21:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:21:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:21:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:21:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:21:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:21:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:21:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:21:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:21:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:21:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:21:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:21:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:21:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:21:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:21:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:21:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:21:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:21:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:21:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:21:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:21:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:21:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:21:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:21:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:21:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:21:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:21:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:21:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:21:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:21:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:21:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:21:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:21:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:21:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:21:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:21:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:21:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:21:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:21:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:21:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:21:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:22:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:22:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:22:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:22:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:22:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:22:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:22:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:22:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:22:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:22:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:22:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:22:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:22:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:22:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:22:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:22:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:22:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:22:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:22:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:22:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:22:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:22:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:22:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:22:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:22:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:22:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:22:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:22:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:22:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:22:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:22:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:22:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:22:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:22:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:22:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:22:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:22:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:22:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:22:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:22:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:22:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:22:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:22:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:22:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:22:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:22:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:22:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:22:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:22:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:22:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:22:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:22:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:22:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:22:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:22:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:22:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:22:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:22:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:22:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:22:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:22:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:22:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:22:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:22:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:22:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:22:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:22:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:22:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:22:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:22:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:22:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:22:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:22:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:23:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:23:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:23:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:23:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:23:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:23:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:23:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:23:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:23:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:23:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:23:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:23:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:23:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:23:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:23:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:23:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:23:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:23:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:23:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:23:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:23:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:23:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:23:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:23:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:23:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:23:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:23:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:23:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:23:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:23:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:23:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:23:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:23:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:23:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:23:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:23:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:23:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:23:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:23:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:23:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:23:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:23:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:23:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:23:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:23:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:23:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:23:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:23:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:23:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:23:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:23:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:23:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:23:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:23:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:23:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:23:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:23:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:23:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:23:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:23:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:23:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:23:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:23:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:23:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:23:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:23:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:23:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:23:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:23:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:24:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:24:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:24:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:24:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:24:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:24:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:24:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:24:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:24:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:24:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:24:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:24:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:24:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:24:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:24:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:24:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:24:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:24:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:24:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:24:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:24:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:24:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:24:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:24:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:24:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:24:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:24:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:24:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:24:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:24:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:24:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:24:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:24:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:24:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:24:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:24:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:24:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:24:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:24:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:24:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:24:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:24:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:24:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:24:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:24:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:24:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:24:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:24:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:24:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:24:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:24:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:24:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:24:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:24:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:24:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:24:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:24:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:24:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:24:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:24:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:24:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:24:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:25:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:25:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:25:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:25:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:25:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:25:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:25:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:25:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:25:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:25:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:25:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:25:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:25:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:25:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:25:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:25:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:25:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:25:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:25:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:25:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:25:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:25:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:25:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:25:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:25:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:25:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:25:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:25:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:25:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:25:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:25:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:25:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:25:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:25:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:25:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:25:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:25:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:25:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:25:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:25:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:25:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:25:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:25:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:25:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:25:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:25:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:25:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:25:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:25:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:25:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:25:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:25:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:25:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:25:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:25:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:25:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:25:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:25:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:25:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:25:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:25:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:25:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:25:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:25:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:25:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:25:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:25:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:25:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:25:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:25:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:25:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:25:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:25:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:26:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:26:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:26:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:26:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:26:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:26:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:26:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:26:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:26:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:26:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:26:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:26:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:26:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:26:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:26:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:26:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:26:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:26:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:26:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:26:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:26:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:26:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:26:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:26:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:26:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:26:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:26:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:26:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:26:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:26:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:26:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:26:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:26:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:26:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:26:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:26:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:26:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:26:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:26:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:26:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:26:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:26:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:26:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:26:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:26:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:26:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:26:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:26:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:26:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:26:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:26:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:26:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:26:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:26:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:26:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:26:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:26:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:26:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:26:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:26:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:26:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:26:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:26:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:26:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:26:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:26:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:26:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:26:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:26:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:27:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:27:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:27:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:27:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:27:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:27:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:27:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:27:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:27:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:27:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:27:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:27:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:27:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:27:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:27:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:27:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:27:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:27:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:27:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:27:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:27:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:27:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:27:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:27:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:27:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:27:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:27:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:27:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:27:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:27:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:27:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:27:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:27:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:27:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:27:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:27:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:27:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:27:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:27:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:27:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:27:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:27:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:27:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:27:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:27:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:27:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:27:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:27:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:27:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:27:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:27:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:27:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:27:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:27:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:27:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:27:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:27:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:27:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:27:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:27:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:27:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:27:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:27:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:27:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:27:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:27:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:27:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:27:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:27:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:27:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:27:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:27:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:27:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:28:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:28:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:28:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:28:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:28:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:28:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:28:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:28:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:28:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:28:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:28:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:28:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:28:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:28:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:28:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:28:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:28:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:28:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:28:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:28:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:28:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:28:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:28:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:28:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:28:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:28:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:28:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:28:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:28:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:28:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:28:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:28:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:28:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:28:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:28:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:28:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:28:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:28:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:28:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:28:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:28:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:28:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:28:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:28:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:28:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:28:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:28:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:28:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:28:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:28:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:28:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:28:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:28:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:28:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:28:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:28:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:28:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:28:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:28:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:28:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:28:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:28:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:28:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:28:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:28:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:28:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:28:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:28:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:29:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:29:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:29:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:29:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:29:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:29:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:29:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:29:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:29:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:29:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:29:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:29:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:29:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:29:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:29:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:29:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:29:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:29:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:29:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:29:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:29:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:29:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:29:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:29:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:29:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:29:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:29:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:29:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:29:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:29:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:29:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:29:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:29:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:29:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:29:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:29:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:29:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:29:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:29:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:29:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:29:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:29:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:29:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:29:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:29:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:29:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:29:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:29:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:29:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:29:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:29:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:29:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:29:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:29:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:29:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:29:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:29:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:29:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:29:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:29:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:29:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:29:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:29:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:29:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:29:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:29:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:29:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:29:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:29:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:30:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:30:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:30:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:30:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:30:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:30:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:30:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:30:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:30:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:30:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:30:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:30:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:30:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:30:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:30:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:30:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:30:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:30:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:30:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:30:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:30:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:30:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:30:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:30:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:30:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:30:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:30:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:30:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:30:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:30:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:30:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:30:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:30:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:30:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:30:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:30:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:30:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:30:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:30:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:30:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:30:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:30:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:30:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:30:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:30:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:30:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:30:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:30:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:30:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:30:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:30:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:30:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:30:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:30:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:30:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:30:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:30:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:30:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:30:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:30:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:30:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:30:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:30:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:30:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:30:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:30:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:30:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:30:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:30:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:30:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:30:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:30:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:30:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:31:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:31:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:31:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:31:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:31:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:31:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:31:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:31:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:31:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:31:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:31:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:31:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:31:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:31:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:31:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:31:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:31:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:31:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:31:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:31:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:31:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:31:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:31:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:31:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:31:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:31:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:31:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:31:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:31:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:31:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:31:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:31:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:31:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:31:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:31:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:31:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:31:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:31:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:31:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:31:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:31:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:31:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:31:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:31:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:31:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:31:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:31:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:31:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:31:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:31:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:31:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:31:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:31:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:31:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:31:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:31:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:31:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:31:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:31:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:31:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:31:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:31:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:31:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:31:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:31:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:31:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:31:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:31:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:31:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:32:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:32:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:32:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:32:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:32:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:32:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:32:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:32:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:32:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:32:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:32:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:32:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:32:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:32:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:32:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:32:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:32:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:32:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:32:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:32:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:32:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:32:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:32:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:32:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:32:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:32:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:32:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:32:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:32:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:32:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:32:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:32:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:32:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:32:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:32:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:32:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:32:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:32:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:32:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:32:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:32:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:32:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:32:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:32:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:32:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:32:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:32:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:32:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:32:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:32:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:32:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:32:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:32:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:32:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:32:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:32:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:32:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:32:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:32:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:32:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:32:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:32:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:32:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:32:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:32:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:32:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:32:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:32:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:32:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:32:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:32:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:32:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:32:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:33:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:33:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:33:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:33:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:33:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:33:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:33:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:33:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:33:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:33:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:33:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:33:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:33:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:33:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:33:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:33:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:33:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:33:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:33:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:33:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:33:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:33:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:33:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:33:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:33:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:33:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:33:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:33:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:33:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:33:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:33:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:33:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:33:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:33:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:33:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:33:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:33:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:33:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:33:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:33:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:33:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:33:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:33:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:33:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:33:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:33:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:33:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:33:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:33:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:33:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:33:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:33:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:33:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:33:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:33:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:33:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:33:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:33:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:33:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:33:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:33:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:33:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:33:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:33:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:33:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:33:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:33:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:33:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:33:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:34:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:34:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:34:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:34:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:34:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:34:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:34:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:34:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:34:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:34:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:34:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:34:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:34:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:34:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:34:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:34:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:34:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:34:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:34:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:34:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:34:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:34:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:34:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:34:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:34:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:34:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:34:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:34:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:34:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:34:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:34:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:34:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:34:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:34:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:34:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:34:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:34:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:34:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:34:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:34:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:34:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:34:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:34:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:34:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:34:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:34:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:34:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:34:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:34:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:34:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:34:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:34:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:34:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:34:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:34:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:34:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:34:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:34:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:34:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:34:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:34:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:34:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:34:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:34:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:34:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:34:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:34:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:34:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:34:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:34:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:34:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:35:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:35:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:35:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:35:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:35:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:35:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:35:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:35:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:35:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:35:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:35:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:35:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:35:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:35:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:35:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:35:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:35:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:35:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:35:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:35:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:35:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:35:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:35:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:35:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:35:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:35:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:35:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:35:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:35:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:35:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:35:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:35:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:35:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:35:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:35:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:35:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:35:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:35:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:35:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:35:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:35:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:35:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:35:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:35:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:35:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:35:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:35:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:35:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:35:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:35:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:35:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:35:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:35:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:35:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:35:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:35:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:35:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:35:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:35:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:35:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:35:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:35:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:35:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:35:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:35:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:36:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:36:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:36:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:36:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:36:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:36:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:36:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:36:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:36:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:36:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:36:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:36:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:36:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:36:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:36:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:36:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:36:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:36:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:36:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:36:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:36:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:36:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:36:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:36:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:36:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:36:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:36:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:36:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:36:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:36:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:36:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:36:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:36:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:36:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:36:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:36:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:36:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:36:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:36:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:36:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:36:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:36:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:36:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:36:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:36:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:36:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:36:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:36:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:36:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:36:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:36:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:36:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:36:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:36:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:36:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:36:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:36:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:36:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:36:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:36:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:36:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:36:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:36:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:36:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:36:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:36:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:36:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:36:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:36:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:36:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:36:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:36:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:36:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:36:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:37:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:37:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:37:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:37:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:37:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:37:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:37:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:37:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:37:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:37:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:37:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:37:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:37:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:37:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:37:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:37:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:37:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:37:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:37:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:37:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:37:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:37:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:37:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:37:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:37:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:37:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:37:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:37:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:37:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:37:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:37:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:37:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:37:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:37:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:37:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:37:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:37:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:37:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:37:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:37:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:37:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:37:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:37:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:37:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:37:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:37:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:37:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:37:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:37:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:37:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:37:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:37:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:37:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:37:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:37:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:37:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:37:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:37:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:37:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:37:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:37:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:37:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:37:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:37:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:37:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:37:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:37:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:37:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:37:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:37:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:37:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:37:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:37:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:37:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:38:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:38:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:38:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:38:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:38:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:38:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:38:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:38:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:38:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:38:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:38:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:38:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:38:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:38:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:38:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:38:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:38:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:38:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:38:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:38:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:38:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:38:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:38:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:38:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:38:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:38:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:38:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:38:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:38:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:38:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:38:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:38:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:38:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:38:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:38:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:38:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:38:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:38:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:38:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:38:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:38:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:38:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:38:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:38:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:38:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:38:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:38:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:38:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:38:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:38:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:38:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:38:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:38:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:38:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:38:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:38:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:38:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:38:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:38:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:38:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:38:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:38:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:38:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:38:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:38:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:38:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:38:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:38:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:38:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:38:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:38:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:38:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:38:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:38:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:39:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:39:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:39:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:39:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:39:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:39:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:39:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:39:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:39:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:39:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:39:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:39:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:39:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:39:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:39:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:39:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:39:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:39:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:39:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:39:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:39:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:39:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:39:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:39:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:39:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:39:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:39:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:39:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:39:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:39:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:39:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:39:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:39:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:39:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:39:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:39:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:39:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:39:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:39:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:39:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:39:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:39:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:39:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:39:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:39:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:39:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:39:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:39:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:39:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:39:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:39:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:39:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:39:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:39:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:39:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:39:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:39:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:39:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:39:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:39:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:39:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:39:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:39:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:39:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:39:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:39:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:39:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:39:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:39:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:39:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-19 16:39:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-19 16:39:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:39:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-19 16:39:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-19 16:40:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-19 16:40:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-19 16:40:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-19 16:40:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-19 16:40:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-19 16:40:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-19 16:40:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-19 16:40:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-19 16:40:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:40:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-19 16:40:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:40:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-19 16:40:03 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 16:40:03 INF Re-authenticating Prometheus client 2025-09-19 16:40:03 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 16:40:03 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 16:40:03 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 16:40:03 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 16:40:03 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 22:52:50 INF Adding event: Scenario_benchmark-stop 2025-09-19 23:20:53 INF Stopping Lodemon 2025-09-19 23:20:53 INF Stopping monitoring service...