==================================================================================================== ============================ Pod describe lodemon-6795db79c6-nckhq ============================ ==================================================================================================== Name: lodemon-6795db79c6-nckhq Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Thu, 18 Sep 2025 19:15:48 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6795db79c6 skaffold.dev/run-id=0c3dce84-cb46-49c2-9846-172dcdf9a232 Annotations: Status: Running IP: 10.106.40.162 IPs: IP: 10.106.40.162 Controlled By: ReplicaSet/lodemon-6795db79c6 Containers: lodemon: Container ID: containerd://c7d472cd4a6e00999e76bbf77403c94d030dd9038761fd8c01b7428cf0087a57 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Thu, 18 Sep 2025 19:15:49 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m8vdc (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-m8vdc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-6795db79c6-nckhq ============================== ==================================================================================================== 2025-09-18 19:15:49 INF Starting lodemon server on port 8080 2025-09-18 19:15:49 INF Initializing clients... 2025-09-18 19:15:49 DBG Fetching ID token for accessing prometheus 2025-09-18 19:15:49 DBG Fetched ID token successfully 2025-09-18 19:15:49 INF Clients initialized successfully 2025-09-18 19:16:24 INF Configuring Lodemon 2025-09-18 19:16:24 INF Configuring monitoring units 2025-09-18 19:16:24 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-18 19:16:24 TRC Parsing Prometheus queries from config 2025-09-18 19:16:24 TRC Parsing component query: DS Java heap mem usage 2025-09-18 19:16:24 TRC Parsing component query: IDM Java heap mem usage 2025-09-18 19:16:24 TRC Parsing component query: AM Java heap mem usage 2025-09-18 19:16:24 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-18 19:16:24 TRC Parsing global query: 2025-09-18 19:16:24 TRC Parsing scoped query: DS Identity backend connections 2025-09-18 19:16:24 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-18 19:16:24 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-18 19:16:24 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-18 19:16:24 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-18 19:16:24 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-18 19:16:24 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-18 19:16:24 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-18 19:16:24 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-18 19:16:24 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-18 19:16:24 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-18 19:16:24 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-18 19:16:24 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-18 19:16:24 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-18 19:16:24 INF  - All monitoring units configured 2025-09-18 19:16:24 INF Configuring monitoring service 2025-09-18 19:16:24 INF  - Adding monitoring units to the service 2025-09-18 19:16:24 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-18 19:16:24 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-18 19:16:24 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-18 19:16:24 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-18 19:16:24 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-18 19:16:24 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-18 19:16:24 INF  - All monitoring units added to the service 2025-09-18 19:16:24 INF Starting Lodemon now 2025-09-18 19:16:24 INF Starting monitoring service... 2025-09-18 19:16:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:16:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:16:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:16:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:16:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:16:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:16:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:16:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:16:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:16:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:16:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:16:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:16:27 INF Adding event: Scenario_benchmark 2025-09-18 19:16:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:16:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:16:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:16:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:16:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:16:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:16:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:16:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:16:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:16:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:16:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:16:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:16:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:16:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:16:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:16:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:16:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:16:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:16:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:16:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:16:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:16:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:16:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:16:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:16:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:16:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:16:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:16:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:16:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:16:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:16:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:16:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:16:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:16:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:16:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:16:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:16:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:16:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:16:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:17:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:17:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:17:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:17:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:17:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:17:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:17:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:17:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:17:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:17:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:17:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:17:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:17:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:17:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:17:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:17:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:17:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:17:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:17:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:17:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:17:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:17:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:17:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:17:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:17:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:17:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:17:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:17:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:17:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:17:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:17:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:17:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:17:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:17:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:17:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:17:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:17:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:17:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:17:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:17:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:17:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:17:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:17:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:17:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:17:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:17:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:17:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:17:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:17:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:17:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:17:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:17:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:17:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:17:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:17:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:17:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:17:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:17:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:17:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:17:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:17:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:17:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:17:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:17:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:17:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:17:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:17:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:17:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:17:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:17:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:17:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:17:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:17:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:17:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:18:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:18:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:18:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:18:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:18:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:18:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:18:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:18:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:18:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:18:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:18:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:18:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:18:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:18:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:18:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:18:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:18:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:18:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:18:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:18:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:18:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:18:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:18:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:18:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:18:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:18:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:18:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:18:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:18:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:18:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:18:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:18:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:18:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:18:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:18:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:18:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:18:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:18:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:18:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:18:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:18:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:18:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:18:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:18:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:18:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:18:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:18:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:18:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:18:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:18:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:18:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:18:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:18:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:18:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:18:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:18:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:18:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:18:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:18:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:18:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:18:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:18:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:18:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:18:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:18:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:18:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:18:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:18:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:18:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:18:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:18:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:18:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:18:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:19:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:19:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:19:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:19:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:19:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:19:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:19:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:19:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:19:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:19:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:19:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:19:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:19:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:19:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:19:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:19:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:19:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:19:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:19:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:19:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:19:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:19:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:19:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:19:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:19:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:19:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:19:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:19:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:19:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:19:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:19:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:19:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:19:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:19:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:19:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:19:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:19:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:19:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:19:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:19:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:19:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:19:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:19:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:19:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:19:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:19:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:19:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:19:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:19:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:19:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:19:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:19:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:19:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:19:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:19:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:19:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:19:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:19:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:19:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:19:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:19:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:19:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:19:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:19:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:19:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:19:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:19:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:19:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:19:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:20:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:20:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:20:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:20:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:20:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:20:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:20:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:20:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:20:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:20:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:20:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:20:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:20:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:20:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:20:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:20:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:20:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:20:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:20:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:20:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:20:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:20:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:20:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:20:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:20:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:20:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:20:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:20:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:20:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:20:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:20:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:20:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:20:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:20:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:20:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:20:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:20:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:20:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:20:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:20:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:20:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:20:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:20:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:20:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:20:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:20:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:20:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:20:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:20:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:20:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:20:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:20:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:20:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:20:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:20:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:20:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:20:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:20:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:20:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:20:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:20:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:20:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:20:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:20:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:20:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:20:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:20:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:20:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:20:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:20:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:20:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:20:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:20:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:21:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:21:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:21:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:21:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:21:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:21:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:21:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:21:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:21:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:21:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:21:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:21:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:21:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:21:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:21:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:21:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:21:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:21:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:21:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:21:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:21:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:21:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:21:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:21:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:21:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:21:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:21:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:21:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:21:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:21:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:21:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:21:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:21:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:21:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:21:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:21:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:21:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:21:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:21:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:21:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:21:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:21:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:21:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:21:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:21:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:21:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:21:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:21:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:21:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:21:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:21:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:21:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:21:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:21:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:21:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:21:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:21:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:21:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:21:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:21:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:21:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:21:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:21:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:21:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:21:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:21:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:21:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:21:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:22:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:22:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:22:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:22:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:22:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:22:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:22:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:22:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:22:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:22:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:22:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:22:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:22:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:22:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:22:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:22:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:22:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:22:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:22:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:22:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:22:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:22:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:22:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:22:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:22:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:22:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:22:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:22:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:22:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:22:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:22:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:22:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:22:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:22:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:22:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:22:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:22:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:22:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:22:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:22:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:22:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:22:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:22:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:22:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:22:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:22:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:22:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:22:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:22:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:22:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:22:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:22:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:22:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:22:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:22:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:22:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:22:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:22:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:22:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:22:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:22:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:22:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:22:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:22:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:22:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:22:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:22:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:22:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:23:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:23:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:23:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:23:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:23:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:23:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:23:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:23:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:23:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:23:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:23:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:23:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:23:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:23:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:23:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:23:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:23:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:23:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:23:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:23:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:23:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:23:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:23:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:23:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:23:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:23:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:23:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:23:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:23:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:23:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:23:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:23:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:23:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:23:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:23:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:23:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:23:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:23:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:23:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:23:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:23:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:23:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:23:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:23:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:23:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:23:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:23:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:23:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:23:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:23:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:23:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:23:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:23:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:23:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:23:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:23:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:23:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:23:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:23:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:23:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:23:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:23:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:23:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:23:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:23:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:23:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:23:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:23:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:23:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:24:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:24:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:24:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:24:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:24:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:24:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:24:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:24:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:24:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:24:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:24:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:24:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:24:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:24:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:24:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:24:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:24:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:24:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:24:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:24:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:24:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:24:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:24:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:24:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:24:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:24:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:24:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:24:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:24:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:24:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:24:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:24:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:24:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:24:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:24:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:24:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:24:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:24:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:24:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:24:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:24:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:24:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:24:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:24:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:24:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:24:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:24:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:24:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:24:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:24:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:24:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:24:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:24:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:24:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:24:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:24:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:24:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:24:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:24:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:24:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:24:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:25:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:25:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:25:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:25:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:25:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:25:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:25:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:25:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:25:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:25:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:25:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:25:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:25:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:25:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:25:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:25:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:25:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:25:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:25:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:25:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:25:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:25:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:25:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:25:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:25:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:25:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:25:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:25:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:25:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:25:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:25:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:25:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:25:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:25:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:25:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:25:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:25:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:25:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:25:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:25:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:25:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:25:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:25:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:25:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:25:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:25:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:25:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:25:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:25:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:25:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:25:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:25:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:25:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:25:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:25:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:25:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:25:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:25:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:25:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:25:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:25:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:25:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:25:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:25:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:25:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:25:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:25:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:25:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:26:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:26:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:26:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:26:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:26:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:26:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:26:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:26:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:26:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:26:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:26:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:26:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:26:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:26:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:26:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:26:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:26:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:26:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:26:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:26:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:26:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:26:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:26:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:26:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:26:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:26:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:26:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:26:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:26:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:26:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:26:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:26:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:26:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:26:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:26:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:26:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:26:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:26:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:26:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:26:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:26:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:26:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:26:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:26:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:26:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:26:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:26:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:26:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:26:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:26:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:26:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:26:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:26:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:26:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:26:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:26:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:26:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:26:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:26:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:26:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:26:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:26:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:26:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:26:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:26:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:26:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:26:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:26:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:26:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:26:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:26:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:26:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:26:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:26:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:27:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:27:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:27:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:27:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:27:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:27:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:27:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:27:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:27:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:27:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:27:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:27:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:27:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:27:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:27:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:27:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:27:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:27:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:27:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:27:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:27:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:27:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:27:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:27:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:27:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:27:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:27:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:27:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:27:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:27:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:27:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:27:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:27:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:27:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:27:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:27:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:27:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:27:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:27:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:27:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:27:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:27:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:27:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:27:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:27:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:27:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:27:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:27:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:27:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:27:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:27:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:27:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:27:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:27:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:27:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:27:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:27:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:27:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:27:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:27:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:27:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:27:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:27:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:27:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:27:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:27:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:27:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:27:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:27:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:27:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:27:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:27:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:27:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:27:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:28:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:28:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:28:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:28:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:28:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:28:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:28:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:28:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:28:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:28:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:28:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:28:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:28:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:28:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:28:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:28:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:28:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:28:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:28:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:28:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:28:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:28:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:28:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:28:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:28:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:28:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:28:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:28:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:28:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:28:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:28:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:28:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:28:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:28:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:28:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:28:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:28:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:28:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:28:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:28:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:28:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:28:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:28:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:28:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:28:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:28:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:28:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:28:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:28:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:28:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:28:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:28:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:28:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:28:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:28:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:28:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:28:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:28:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:28:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:28:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:28:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:28:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:28:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:28:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:28:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:28:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:28:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:28:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:28:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:28:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:28:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:28:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:28:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:28:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:29:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:29:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:29:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:29:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:29:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:29:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:29:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:29:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:29:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:29:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:29:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:29:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:29:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:29:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:29:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:29:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:29:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:29:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:29:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:29:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:29:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:29:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:29:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:29:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:29:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:29:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:29:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:29:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:29:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:29:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:29:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:29:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:29:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:29:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:29:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:29:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:29:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:29:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:29:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:29:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:29:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:29:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:29:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:29:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:29:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:29:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:29:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:29:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:29:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:29:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:29:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:29:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:29:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:29:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:29:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:29:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:29:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:29:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:29:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:29:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:29:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:29:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:29:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:29:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:29:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:29:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:29:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:29:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:29:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:30:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:30:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:30:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:30:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:30:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:30:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:30:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:30:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:30:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:30:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:30:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:30:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:30:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:30:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:30:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:30:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:30:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:30:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:30:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:30:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:30:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:30:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:30:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:30:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:30:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:30:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:30:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:30:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:30:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:30:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:30:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:30:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:30:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:30:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:30:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:30:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:30:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:30:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:30:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:30:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:30:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:30:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:30:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:30:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:30:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:30:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:30:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:30:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:30:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:30:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:30:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:30:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:30:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:30:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:30:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:30:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:30:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:30:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:30:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:30:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:30:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:30:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:30:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:30:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:30:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:30:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:30:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:30:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:30:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:30:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:30:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:30:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:30:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:31:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:31:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:31:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:31:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:31:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:31:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:31:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:31:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:31:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:31:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:31:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:31:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:31:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:31:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:31:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:31:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:31:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:31:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:31:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:31:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:31:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:31:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:31:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:31:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:31:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:31:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:31:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:31:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:31:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:31:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:31:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:31:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:31:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:31:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:31:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:31:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:31:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:31:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:31:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:31:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:31:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:31:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:31:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:31:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:31:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:31:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:31:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:31:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:31:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:31:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:31:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:31:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:31:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:31:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:31:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:31:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:31:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:31:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:31:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:31:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:31:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:31:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:31:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:31:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:31:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:31:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:31:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:31:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:31:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:32:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:32:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:32:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:32:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:32:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:32:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:32:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:32:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:32:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:32:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:32:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:32:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:32:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:32:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:32:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:32:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:32:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:32:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:32:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:32:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:32:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:32:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:32:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:32:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:32:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:32:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:32:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:32:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:32:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:32:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:32:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:32:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:32:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:32:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:32:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:32:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:32:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:32:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:32:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:32:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:32:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:32:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:32:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:32:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:32:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:32:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:32:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:32:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:32:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:32:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:32:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:32:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:32:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:32:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:32:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:32:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:32:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:32:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:32:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:32:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:32:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:32:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:32:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:32:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:32:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:32:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:32:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:32:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:32:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:32:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:32:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:32:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:32:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:32:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:33:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:33:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:33:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:33:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:33:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:33:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:33:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:33:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:33:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:33:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:33:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:33:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:33:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:33:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:33:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:33:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:33:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:33:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:33:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:33:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:33:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:33:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:33:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:33:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:33:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:33:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:33:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:33:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:33:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:33:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:33:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:33:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:33:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:33:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:33:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:33:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:33:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:33:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:33:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:33:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:33:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:33:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:33:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:33:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:33:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:33:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:33:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:33:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:33:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:33:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:33:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:33:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:33:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:33:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:33:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:33:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:33:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:33:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:33:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:33:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:33:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:33:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:33:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:33:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:33:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:33:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:33:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:33:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:33:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:33:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:33:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:33:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:33:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:34:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:34:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:34:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:34:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:34:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:34:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:34:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:34:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:34:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:34:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:34:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:34:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:34:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:34:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:34:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:34:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:34:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:34:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:34:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:34:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:34:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:34:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:34:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:34:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:34:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:34:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:34:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:34:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:34:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:34:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:34:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:34:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:34:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:34:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:34:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:34:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:34:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:34:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:34:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:34:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:34:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:34:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:34:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:34:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:34:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:34:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:34:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:34:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:34:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:34:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:34:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:34:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:34:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:34:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:34:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:34:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:34:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:34:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:34:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:34:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:34:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:34:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:34:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:34:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:34:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:34:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:34:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:34:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:35:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:35:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:35:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:35:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:35:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:35:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:35:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:35:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:35:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:35:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:35:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:35:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:35:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:35:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:35:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:35:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:35:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:35:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:35:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:35:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:35:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:35:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:35:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:35:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:35:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:35:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:35:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:35:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:35:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:35:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:35:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:35:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:35:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:35:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:35:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:35:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:35:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:35:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:35:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:35:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:35:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:35:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:35:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:35:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:35:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:35:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:35:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:35:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:35:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:35:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:35:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:35:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:35:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:35:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:35:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:35:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:35:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:35:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:35:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:35:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:35:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:35:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:35:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:35:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:35:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:35:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:35:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:35:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:35:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:36:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:36:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:36:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:36:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:36:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:36:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:36:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:36:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:36:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:36:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:36:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:36:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:36:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:36:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:36:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:36:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:36:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:36:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:36:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:36:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:36:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:36:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:36:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:36:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:36:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:36:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:36:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:36:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:36:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:36:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:36:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:36:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:36:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:36:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:36:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:36:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:36:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:36:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:36:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:36:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:36:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:36:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:36:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:36:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:36:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:36:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:36:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:36:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:36:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:36:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:36:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:36:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:36:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:36:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:36:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:36:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:36:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:36:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:36:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:36:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:36:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:36:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:36:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:36:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:36:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:36:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:36:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:36:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:36:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:36:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:36:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:36:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:36:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:37:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:37:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:37:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:37:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:37:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:37:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:37:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:37:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:37:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:37:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:37:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:37:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:37:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:37:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:37:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:37:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:37:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:37:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:37:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:37:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:37:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:37:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:37:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:37:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:37:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:37:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:37:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:37:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:37:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:37:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:37:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:37:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:37:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:37:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:37:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:37:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:37:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:37:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:37:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:37:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:37:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:37:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:37:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:37:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:37:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:37:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:37:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:37:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:37:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:37:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:37:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:37:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:37:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:37:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:37:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:37:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:37:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:37:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:37:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:37:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:37:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:37:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:37:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:37:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:37:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:37:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:37:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:37:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:37:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:38:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:38:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:38:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:38:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:38:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:38:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:38:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:38:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:38:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:38:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:38:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:38:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:38:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:38:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:38:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:38:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:38:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:38:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:38:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:38:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:38:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:38:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:38:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:38:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:38:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:38:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:38:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:38:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:38:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:38:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:38:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:38:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:38:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:38:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:38:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:38:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:38:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:38:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:38:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:38:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:38:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:38:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:38:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:38:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:38:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:38:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:38:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:38:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:38:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:38:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:38:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:38:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:38:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:38:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:38:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:38:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:38:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:38:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:38:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:38:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:38:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:38:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:39:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:39:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:39:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:39:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:39:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:39:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:39:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:39:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:39:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:39:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:39:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:39:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:39:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:39:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:39:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:39:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:39:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:39:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:39:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:39:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:39:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:39:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:39:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:39:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:39:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:39:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:39:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:39:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:39:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:39:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:39:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:39:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:39:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:39:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:39:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:39:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:39:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:39:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:39:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:39:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:39:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:39:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:39:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:39:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:39:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:39:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:39:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:39:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:39:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:39:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:39:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:39:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:39:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:39:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:39:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:39:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:39:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:39:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:39:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:39:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:39:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:39:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:39:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:39:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:39:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:39:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:39:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:39:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:39:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:39:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:39:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:39:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:39:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:39:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:40:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:40:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:40:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:40:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:40:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:40:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:40:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:40:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:40:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:40:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:40:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:40:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:40:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:40:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:40:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:40:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:40:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:40:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:40:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:40:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:40:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:40:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:40:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:40:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:40:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:40:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:40:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:40:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:40:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:40:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:40:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:40:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:40:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:40:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:40:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:40:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:40:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:40:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:40:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:40:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:40:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:40:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:40:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:40:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:40:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:40:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:40:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:40:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:40:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:40:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:40:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:40:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:40:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:40:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:40:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:40:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:40:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:40:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:40:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:40:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:40:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:40:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:40:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:40:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:40:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:40:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:40:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:40:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:40:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:40:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:40:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:40:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:40:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:40:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:41:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:41:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:41:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:41:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:41:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:41:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:41:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:41:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:41:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:41:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:41:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:41:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:41:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:41:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:41:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:41:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:41:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:41:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:41:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:41:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:41:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:41:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:41:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:41:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:41:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:41:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:41:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:41:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:41:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:41:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:41:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:41:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:41:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:41:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:41:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:41:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:41:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:41:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:41:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:41:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:41:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:41:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:41:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:41:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:41:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:41:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:41:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:41:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:41:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:41:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:41:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:41:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:41:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:41:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:41:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:41:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:41:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:41:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:41:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:41:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:41:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:41:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:41:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:41:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:41:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:41:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:41:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:41:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:41:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:41:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:41:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:41:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:41:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:41:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:42:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:42:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:42:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:42:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:42:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:42:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:42:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:42:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:42:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:42:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:42:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:42:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:42:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:42:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:42:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:42:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:42:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:42:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:42:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:42:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:42:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:42:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:42:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:42:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:42:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:42:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:42:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:42:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:42:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:42:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:42:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:42:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:42:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:42:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:42:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:42:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:42:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:42:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:42:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:42:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:42:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:42:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:42:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:42:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:42:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:42:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:42:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:42:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:42:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:42:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:42:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:42:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:42:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:42:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:42:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:42:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:42:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:42:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:42:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:42:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:42:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:42:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:42:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:42:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:42:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:42:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:42:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:42:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:42:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:42:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:42:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:42:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:42:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:42:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:43:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:43:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:43:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:43:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:43:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:43:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:43:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:43:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:43:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:43:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:43:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:43:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:43:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:43:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:43:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:43:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:43:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:43:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:43:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:43:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:43:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:43:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:43:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:43:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:43:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:43:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:43:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:43:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:43:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:43:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:43:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:43:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:43:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:43:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:43:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:43:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:43:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:43:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:43:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:43:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:43:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:43:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:43:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:43:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:43:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:43:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:43:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:43:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:43:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:43:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:43:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:43:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:43:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:43:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:43:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:43:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:43:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:43:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:43:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:43:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:43:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:43:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:43:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:43:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:43:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:43:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:43:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:43:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:43:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:43:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:43:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:43:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:43:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:44:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:44:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:44:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:44:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:44:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:44:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:44:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:44:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:44:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:44:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:44:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:44:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:44:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:44:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:44:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:44:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:44:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:44:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:44:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:44:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:44:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:44:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:44:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:44:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:44:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:44:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:44:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:44:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:44:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:44:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:44:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:44:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:44:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:44:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:44:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:44:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:44:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:44:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:44:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:44:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:44:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:44:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:44:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:44:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:44:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:44:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:44:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:44:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:44:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:44:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:44:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:44:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:44:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:44:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:44:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:44:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:44:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:44:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:44:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:44:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:44:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:44:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:44:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:44:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:44:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:44:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:44:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:44:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:44:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:45:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:45:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:45:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:45:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:45:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:45:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:45:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:45:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:45:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:45:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:45:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:45:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:45:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:45:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:45:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:45:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:45:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:45:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:45:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:45:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:45:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:45:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:45:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:45:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:45:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:45:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:45:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:45:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:45:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:45:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:45:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:45:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:45:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:45:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:45:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:45:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:45:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:45:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:45:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:45:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:45:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:45:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:45:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:45:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:45:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:45:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:45:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:45:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:45:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:45:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:45:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:45:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:45:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:45:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:45:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:45:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:45:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:45:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:45:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:45:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:45:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:45:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:45:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:45:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:45:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:45:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:45:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:45:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:45:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:45:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:45:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:45:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:45:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:45:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:46:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:46:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:46:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:46:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:46:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:46:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:46:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:46:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:46:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:46:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:46:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:46:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:46:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:46:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:46:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:46:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:46:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:46:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:46:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:46:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:46:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:46:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:46:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:46:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:46:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:46:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:46:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:46:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:46:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:46:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:46:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:46:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:46:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:46:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:46:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:46:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:46:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:46:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:46:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:46:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:46:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:46:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:46:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:46:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:46:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:46:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:46:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:46:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:46:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:46:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:46:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:46:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:46:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:46:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:46:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:46:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:46:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:46:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:46:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:46:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:46:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:46:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:46:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:46:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:46:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:46:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:46:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:46:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:46:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:46:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:46:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:46:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:46:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:46:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:47:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:47:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:47:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:47:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:47:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:47:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:47:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:47:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:47:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:47:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:47:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:47:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:47:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:47:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:47:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:47:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:47:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:47:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:47:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:47:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:47:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:47:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:47:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:47:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:47:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:47:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:47:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:47:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:47:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:47:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:47:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:47:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:47:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:47:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:47:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:47:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:47:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:47:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:47:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:47:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:47:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:47:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:47:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:47:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:47:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:47:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:47:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:47:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:47:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:47:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:47:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:47:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:47:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:47:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:47:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:47:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:47:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:47:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:47:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:47:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:47:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:47:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:47:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:47:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:47:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:47:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:47:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:47:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:47:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:47:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:47:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:47:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:47:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:48:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:48:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:48:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:48:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:48:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:48:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:48:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:48:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:48:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:48:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:48:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:48:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:48:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:48:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:48:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:48:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:48:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:48:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:48:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:48:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:48:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:48:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:48:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:48:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:48:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:48:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:48:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:48:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:48:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:48:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:48:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:48:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:48:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:48:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:48:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:48:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:48:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:48:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:48:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:48:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:48:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:48:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:48:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:48:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:48:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:48:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:48:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:48:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:48:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:48:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:48:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:48:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:48:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:48:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:48:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:48:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:48:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:48:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:48:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:48:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:48:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:48:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:48:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:48:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:48:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:48:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:48:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:48:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:49:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:49:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:49:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:49:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:49:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:49:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:49:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:49:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:49:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:49:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:49:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:49:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:49:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:49:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:49:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:49:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:49:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:49:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:49:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:49:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:49:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:49:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:49:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:49:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:49:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:49:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:49:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:49:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:49:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:49:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:49:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:49:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:49:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:49:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:49:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:49:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:49:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:49:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:49:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:49:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:49:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:49:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:49:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:49:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:49:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:49:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:49:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:49:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:49:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:49:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:49:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:49:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:49:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:49:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:49:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:49:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:49:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:49:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:49:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:49:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:49:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:49:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:49:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:49:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:49:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:49:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:49:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:49:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:50:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:50:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:50:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:50:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:50:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:50:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:50:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:50:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:50:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:50:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:50:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:50:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:50:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:50:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:50:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:50:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:50:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:50:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:50:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:50:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:50:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:50:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:50:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:50:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:50:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:50:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:50:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:50:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:50:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:50:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:50:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:50:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:50:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:50:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:50:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:50:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:50:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:50:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:50:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:50:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:50:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:50:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:50:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:50:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:50:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:50:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:50:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:50:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:50:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:50:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:50:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:50:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:50:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:50:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:50:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:50:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:50:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:50:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:50:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:50:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:50:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:50:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:50:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:50:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:50:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:50:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:50:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:50:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:51:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:51:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:51:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:51:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:51:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:51:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:51:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:51:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:51:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:51:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:51:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:51:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:51:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:51:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:51:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:51:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:51:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:51:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:51:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:51:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:51:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:51:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:51:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:51:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:51:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:51:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:51:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:51:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:51:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:51:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:51:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:51:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:51:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:51:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:51:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:51:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:51:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:51:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:51:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:51:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:51:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:51:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:51:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:51:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:51:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:51:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:51:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:51:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:51:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:51:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:51:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:51:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:51:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:51:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:51:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:51:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:51:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:51:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:51:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:51:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:51:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:51:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:51:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:51:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:51:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:51:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:51:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:51:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:52:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:52:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:52:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:52:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:52:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:52:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:52:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:52:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:52:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:52:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:52:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:52:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:52:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:52:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:52:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:52:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:52:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:52:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:52:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:52:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:52:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:52:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:52:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:52:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:52:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:52:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:52:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:52:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:52:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:52:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:52:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:52:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:52:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:52:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:52:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:52:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:52:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:52:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:52:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:52:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:52:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:52:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:52:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:52:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:52:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:52:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:52:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:52:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:52:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:52:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:52:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:52:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:52:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:52:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:52:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:52:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:52:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:52:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:52:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:52:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:52:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:52:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:52:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:52:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:52:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:52:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:52:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:52:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:53:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:53:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:53:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:53:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:53:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:53:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:53:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:53:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:53:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:53:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:53:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:53:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:53:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:53:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:53:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:53:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:53:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:53:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:53:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:53:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:53:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:53:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:53:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:53:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:53:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:53:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:53:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:53:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:53:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:53:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:53:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:53:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:53:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:53:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:53:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:53:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:53:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:53:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:53:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:53:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:53:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:53:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:53:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:53:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:53:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:53:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:53:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:53:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:53:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:53:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:53:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:53:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:53:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:53:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:53:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:53:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:53:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:53:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:53:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:53:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:53:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:53:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:53:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:53:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:53:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:53:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:53:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:53:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:53:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:54:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:54:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:54:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:54:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:54:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:54:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:54:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:54:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:54:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:54:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:54:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:54:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:54:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:54:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:54:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:54:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:54:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:54:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:54:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:54:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:54:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:54:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:54:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:54:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:54:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:54:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:54:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:54:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:54:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:54:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:54:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:54:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:54:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:54:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:54:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:54:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:54:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:54:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:54:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:54:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:54:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:54:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:54:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:54:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:54:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:54:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:54:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:54:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:54:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:54:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:54:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:54:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:54:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:54:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:54:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:54:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:54:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:54:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:54:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:54:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:54:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:55:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:55:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:55:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:55:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:55:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:55:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:55:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:55:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:55:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:55:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:55:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:55:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:55:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:55:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:55:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:55:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:55:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:55:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:55:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:55:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:55:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:55:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:55:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:55:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:55:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:55:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:55:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:55:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:55:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:55:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:55:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:55:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:55:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:55:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:55:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:55:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:55:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:55:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:55:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:55:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:55:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:55:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:55:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:55:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:55:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:55:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:55:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:55:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:55:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:55:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:55:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:55:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:55:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:55:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:55:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:55:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:55:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:55:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:55:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:55:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:55:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:55:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:55:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:55:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:55:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:55:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:55:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:55:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:56:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:56:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:56:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:56:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:56:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:56:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:56:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:56:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:56:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:56:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:56:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:56:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:56:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:56:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:56:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:56:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:56:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:56:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:56:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:56:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:56:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:56:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:56:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:56:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:56:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:56:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:56:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:56:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:56:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:56:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:56:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:56:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:56:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:56:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:56:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:56:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:56:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:56:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:56:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:56:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:56:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:56:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:56:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:56:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:56:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:56:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:56:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:56:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:56:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:56:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:56:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:56:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:56:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:56:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:56:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:56:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:56:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:56:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:56:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:56:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:56:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:56:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:56:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:56:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:56:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:56:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:56:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:56:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:56:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:57:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:57:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:57:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:57:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:57:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:57:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:57:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:57:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:57:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:57:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:57:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:57:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:57:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:57:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:57:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:57:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:57:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:57:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:57:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:57:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:57:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:57:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:57:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:57:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:57:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:57:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:57:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:57:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:57:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:57:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:57:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:57:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:57:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:57:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:57:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:57:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:57:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:57:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:57:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:57:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:57:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:57:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:57:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:57:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:57:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:57:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:57:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:57:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:57:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:57:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:57:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:57:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:57:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:57:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:57:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:57:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:57:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:57:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:57:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:57:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:57:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:57:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:57:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:57:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:57:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:57:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:57:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:57:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:57:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:57:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:57:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:57:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:57:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:58:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:58:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:58:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:58:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:58:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:58:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:58:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:58:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:58:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:58:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:58:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:58:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:58:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:58:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:58:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:58:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:58:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:58:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:58:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:58:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:58:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:58:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:58:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:58:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:58:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:58:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:58:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:58:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:58:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:58:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:58:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:58:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:58:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:58:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:58:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:58:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:58:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:58:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:58:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:58:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:58:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:58:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:58:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:58:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:58:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:58:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:58:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:58:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:58:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:58:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:58:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:58:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:58:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:58:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:58:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:58:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:58:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:58:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:58:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:58:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:58:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:58:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:58:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:58:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:58:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:58:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:58:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:58:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:58:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:59:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:59:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:59:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:59:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:59:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:59:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:59:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:59:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:59:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:59:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:59:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:59:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:59:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:59:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:59:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:59:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:59:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:59:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:59:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:59:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:59:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:59:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:59:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:59:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:59:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:59:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:59:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:59:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:59:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:59:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:59:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:59:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:59:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:59:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:59:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:59:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:59:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:59:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:59:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:59:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:59:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:59:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:59:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:59:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:59:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:59:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:59:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:59:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:59:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:59:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:59:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:59:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:59:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:59:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:59:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:59:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:59:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:59:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:59:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 19:59:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 19:59:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 19:59:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 19:59:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 19:59:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 19:59:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 19:59:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 19:59:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 19:59:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 19:59:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 19:59:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 19:59:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 19:59:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 19:59:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:00:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:00:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:00:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:00:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:00:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:00:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:00:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:00:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:00:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:00:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:00:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:00:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:00:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:00:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:00:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:00:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:00:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:00:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:00:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:00:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:00:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:00:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:00:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:00:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:00:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:00:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:00:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:00:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:00:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:00:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:00:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:00:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:00:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:00:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:00:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:00:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:00:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:00:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:00:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:00:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:00:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:00:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:00:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:00:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:00:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:00:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:00:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:00:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:00:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:00:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:00:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:00:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:00:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:00:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:00:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:00:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:00:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:00:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:00:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:00:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:00:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:00:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:00:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:00:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:00:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:00:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:00:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:00:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:01:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:01:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:01:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:01:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:01:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:01:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:01:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:01:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:01:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:01:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:01:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:01:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:01:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:01:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:01:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:01:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:01:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:01:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:01:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:01:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:01:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:01:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:01:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:01:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:01:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:01:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:01:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:01:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:01:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:01:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:01:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:01:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:01:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:01:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:01:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:01:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:01:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:01:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:01:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:01:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:01:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:01:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:01:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:01:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:01:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:01:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:01:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:01:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:01:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:01:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:01:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:01:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:01:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:01:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:01:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:01:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:01:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:01:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:01:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:01:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:01:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:01:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:01:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:01:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:01:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:01:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:01:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:01:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:01:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:02:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:02:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:02:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:02:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:02:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:02:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:02:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:02:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:02:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:02:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:02:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:02:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:02:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:02:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:02:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:02:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:02:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:02:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:02:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:02:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:02:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:02:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:02:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:02:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:02:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:02:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:02:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:02:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:02:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:02:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:02:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:02:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:02:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:02:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:02:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:02:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:02:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:02:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:02:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:02:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:02:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:02:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:02:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:02:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:02:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:02:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:02:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:02:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:02:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:02:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:02:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:02:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:02:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:02:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:02:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:02:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:02:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:02:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:02:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:02:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:02:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:02:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:02:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:02:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:02:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:02:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:02:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:02:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:02:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:02:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:02:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:02:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:02:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:02:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:03:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:03:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:03:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:03:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:03:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:03:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:03:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:03:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:03:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:03:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:03:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:03:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:03:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:03:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:03:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:03:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:03:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:03:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:03:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:03:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:03:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:03:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:03:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:03:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:03:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:03:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:03:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:03:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:03:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:03:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:03:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:03:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:03:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:03:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:03:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:03:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:03:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:03:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:03:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:03:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:03:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:03:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:03:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:03:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:03:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:03:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:03:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:03:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:03:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:03:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:03:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:03:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:03:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:03:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:03:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:03:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:03:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:03:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:03:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:03:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:03:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:03:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:03:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:03:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:03:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:03:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:03:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:03:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:03:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:03:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:03:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:03:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:03:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:04:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:04:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:04:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:04:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:04:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:04:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:04:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:04:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:04:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:04:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:04:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:04:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:04:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:04:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:04:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:04:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:04:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:04:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:04:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:04:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:04:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:04:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:04:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:04:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:04:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:04:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:04:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:04:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:04:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:04:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:04:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:04:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:04:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:04:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:04:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:04:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:04:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:04:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:04:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:04:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:04:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:04:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:04:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:04:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:04:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:04:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:04:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:04:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:04:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:04:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:04:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:04:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:04:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:04:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:04:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:04:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:04:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:04:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:04:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:04:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:04:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:04:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:04:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:04:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:04:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:04:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:04:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:04:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:04:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:04:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:04:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:04:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:04:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:04:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:05:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:05:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:05:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:05:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:05:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:05:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:05:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:05:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:05:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:05:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:05:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:05:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:05:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:05:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:05:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:05:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:05:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:05:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:05:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:05:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:05:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:05:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:05:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:05:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:05:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:05:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:05:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:05:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:05:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:05:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:05:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:05:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:05:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:05:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:05:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:05:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:05:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:05:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:05:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:05:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:05:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:05:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:05:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:05:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:05:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:05:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:05:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:05:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:05:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:05:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:05:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:05:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:05:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:05:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:05:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:05:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:05:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:05:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:05:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:05:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:05:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:05:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:05:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:06:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:06:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:06:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:06:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:06:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:06:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:06:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:06:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:06:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:06:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:06:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:06:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:06:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:06:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:06:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:06:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:06:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:06:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:06:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:06:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:06:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:06:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:06:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:06:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:06:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:06:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:06:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:06:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:06:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:06:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:06:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:06:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:06:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:06:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:06:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:06:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:06:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:06:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:06:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:06:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:06:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:06:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:06:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:06:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:06:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:06:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:06:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:06:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:06:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:06:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:06:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:06:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:06:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:06:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:06:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:06:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:06:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:06:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:06:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:06:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:06:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:06:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:06:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:06:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:06:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:06:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:06:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:06:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:06:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:06:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:06:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:06:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:06:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:07:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:07:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:07:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:07:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:07:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:07:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:07:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:07:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:07:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:07:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:07:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:07:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:07:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:07:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:07:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:07:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:07:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:07:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:07:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:07:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:07:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:07:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:07:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:07:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:07:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:07:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:07:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:07:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:07:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:07:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:07:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:07:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:07:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:07:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:07:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:07:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:07:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:07:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:07:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:07:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:07:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:07:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:07:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:07:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:07:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:07:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:07:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:07:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:07:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:07:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:07:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:07:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:07:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:07:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:07:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:07:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:07:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:07:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:07:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:07:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:07:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:07:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:07:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:07:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:07:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:07:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:07:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:07:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:07:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:07:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:07:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:07:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:07:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:07:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:08:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:08:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:08:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:08:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:08:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:08:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:08:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:08:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:08:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:08:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:08:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:08:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:08:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:08:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:08:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:08:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:08:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:08:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:08:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:08:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:08:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:08:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:08:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:08:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:08:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:08:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:08:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:08:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:08:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:08:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:08:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:08:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:08:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:08:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:08:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:08:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:08:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:08:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:08:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:08:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:08:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:08:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:08:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:08:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:08:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:08:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:08:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:08:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:08:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:08:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:08:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:08:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:08:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:08:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:08:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:08:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:08:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:08:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:08:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:08:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:08:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:08:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:08:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:08:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:08:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:08:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:08:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:08:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:08:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:09:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:09:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:09:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:09:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:09:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:09:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:09:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:09:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:09:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:09:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:09:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:09:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:09:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:09:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:09:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:09:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:09:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:09:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:09:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:09:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:09:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:09:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:09:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:09:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:09:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:09:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:09:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:09:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:09:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:09:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:09:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:09:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:09:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:09:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:09:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:09:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:09:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:09:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:09:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:09:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:09:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:09:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:09:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:09:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:09:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:09:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:09:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:09:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:09:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:09:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:09:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:09:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:09:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:09:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:09:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:09:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:09:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:09:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:09:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:09:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:09:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:09:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:09:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:09:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:09:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:09:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:09:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:09:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:09:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:09:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:09:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:09:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:09:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:10:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:10:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:10:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:10:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:10:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:10:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:10:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:10:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:10:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:10:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:10:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:10:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:10:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:10:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:10:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:10:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:10:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:10:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:10:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:10:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:10:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:10:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:10:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:10:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:10:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:10:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:10:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:10:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:10:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:10:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:10:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:10:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:10:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:10:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:10:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:10:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:10:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:10:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:10:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:10:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:10:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:10:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:10:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:10:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:10:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:10:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:10:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:10:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:10:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:10:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:10:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:10:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:10:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:10:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:10:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:10:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:10:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:10:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:10:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:10:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:10:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:10:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:10:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:10:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:10:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:10:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:10:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:10:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:11:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:11:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:11:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:11:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:11:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:11:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:11:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:11:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:11:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:11:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:11:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:11:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:11:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:11:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:11:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:11:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:11:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:11:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:11:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:11:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:11:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:11:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:11:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:11:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:11:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:11:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:11:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:11:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:11:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:11:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:11:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:11:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:11:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:11:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:11:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:11:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:11:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:11:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:11:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:11:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:11:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:11:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:11:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:11:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:11:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:11:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:11:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:11:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:11:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:11:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:11:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:11:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:11:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:11:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:11:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:11:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:11:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:11:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:11:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:11:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:11:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:11:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:11:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:11:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:11:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:11:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:11:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:11:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:12:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:12:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:12:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:12:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:12:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:12:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:12:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:12:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:12:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:12:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:12:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:12:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:12:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:12:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:12:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:12:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:12:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:12:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:12:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:12:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:12:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:12:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:12:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:12:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:12:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:12:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:12:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:12:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:12:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:12:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:12:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:12:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:12:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:12:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:12:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:12:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:12:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:12:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:12:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:12:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:12:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:12:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:12:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:12:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:12:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:12:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:12:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:12:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:12:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:12:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:12:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:12:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:12:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:12:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:12:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:12:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:12:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:12:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:12:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:12:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:12:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:12:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:12:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:12:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:12:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:12:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:12:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:12:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:13:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:13:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:13:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:13:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:13:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:13:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:13:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:13:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:13:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:13:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:13:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:13:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:13:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:13:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:13:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:13:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:13:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:13:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:13:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:13:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:13:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:13:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:13:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:13:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:13:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:13:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:13:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:13:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:13:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:13:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:13:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:13:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:13:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:13:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:13:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:13:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:13:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:13:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:13:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:13:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:13:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:13:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:13:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:13:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:13:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:13:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:13:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:13:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:13:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:13:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:13:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:13:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:13:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:13:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:13:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:13:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:13:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:13:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:13:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:13:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:13:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:13:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:13:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:13:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:13:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:13:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:13:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:13:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:13:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:14:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:14:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:14:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:14:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:14:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:14:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:14:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:14:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:14:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:14:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:14:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:14:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:14:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:14:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:14:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:14:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:14:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:14:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:14:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:14:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:14:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:14:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:14:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:14:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:14:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:14:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:14:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:14:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:14:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:14:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:14:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:14:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:14:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:14:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:14:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:14:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:14:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:14:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:14:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:14:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:14:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:14:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:14:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:14:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:14:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:14:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:14:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:14:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:14:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:14:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:14:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:14:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:14:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:14:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:14:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:14:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:14:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:14:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:14:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:14:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:14:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:14:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:14:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:14:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:14:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:14:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:14:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:14:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:14:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:14:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:14:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:14:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:14:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:15:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:15:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:15:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:15:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:15:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:15:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:15:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:15:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:15:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:15:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:15:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:15:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:15:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:15:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:15:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:15:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:15:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:15:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:15:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:15:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:15:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:15:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:15:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:15:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:15:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:15:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:15:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:15:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:15:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:15:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:15:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:15:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:15:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:15:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:15:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:15:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:15:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:15:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:15:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:15:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:15:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:15:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:15:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:15:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:15:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:15:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:15:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:15:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:15:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:15:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:15:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:15:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:15:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:15:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:15:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:15:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:15:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:15:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:15:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:15:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:15:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:15:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:15:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:15:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:15:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:15:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:15:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:15:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:15:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:15:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:15:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:15:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:15:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:15:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:16:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:16:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:16:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:16:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:16:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:16:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:16:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:16:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:16:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:16:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:16:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:16:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:16:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:16:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:16:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:16:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:16:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:16:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:16:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:16:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:16:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:16:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:16:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:16:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:16:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:16:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:16:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:16:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:16:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:16:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:16:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:16:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:16:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:16:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:16:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:16:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:16:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:16:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:16:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:16:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:16:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:16:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:16:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:16:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:16:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:16:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:16:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:16:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:16:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:16:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:16:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:16:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:16:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:16:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:16:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:16:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:16:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:16:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:16:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:16:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:16:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:16:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:16:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:16:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:16:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:16:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:16:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:16:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:16:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:17:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:17:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:17:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:17:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:17:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:17:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:17:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:17:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:17:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:17:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:17:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:17:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:17:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:17:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:17:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:17:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:17:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:17:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:17:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:17:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:17:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:17:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:17:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:17:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:17:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:17:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:17:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:17:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:17:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:17:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:17:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:17:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:17:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:17:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:17:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:17:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:17:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:17:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:17:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:17:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:17:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:17:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:17:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:17:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:17:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:17:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:17:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:17:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:17:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:17:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:17:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:17:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:17:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:17:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:17:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:17:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:17:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:17:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:17:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:17:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:17:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:17:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:17:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:17:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:17:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:17:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:17:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:17:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:17:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:17:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:17:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:17:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:17:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:17:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:18:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:18:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:18:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:18:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:18:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:18:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:18:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:18:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:18:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:18:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:18:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:18:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:18:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:18:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:18:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:18:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:18:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:18:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:18:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:18:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:18:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:18:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:18:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:18:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:18:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:18:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:18:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:18:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:18:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:18:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:18:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:18:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:18:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:18:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:18:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:18:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:18:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:18:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:18:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:18:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:18:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:18:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:18:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:18:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:18:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:18:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:18:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:18:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:18:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:18:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:18:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:18:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:18:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:18:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:18:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:18:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:18:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:18:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:18:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:18:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:18:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:18:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:18:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:18:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:18:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:18:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:18:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:18:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:18:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:18:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:18:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:18:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:18:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:19:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:19:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:19:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:19:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:19:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:19:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:19:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:19:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:19:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:19:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:19:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:19:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:19:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:19:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:19:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:19:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:19:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:19:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:19:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:19:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:19:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:19:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:19:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:19:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:19:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:19:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:19:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:19:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:19:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:19:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:19:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:19:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:19:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:19:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:19:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:19:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:19:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:19:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:19:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:19:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:19:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:19:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:19:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:19:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:19:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:19:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:19:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:19:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:19:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:19:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:19:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:19:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:19:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:19:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:19:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:19:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:19:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:19:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:19:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:19:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:19:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:19:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:19:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:19:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:19:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:19:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:19:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:19:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:20:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:20:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:20:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:20:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:20:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:20:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:20:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:20:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:20:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:20:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:20:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:20:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:20:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:20:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:20:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:20:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:20:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:20:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:20:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:20:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:20:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:20:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:20:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:20:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:20:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:20:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:20:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:20:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:20:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:20:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:20:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:20:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:20:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:20:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:20:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:20:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:20:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:20:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:20:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:20:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:20:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:20:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:20:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:20:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:20:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:20:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-18 20:20:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-18 20:20:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-18 20:20:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-18 20:20:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-18 20:20:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-18 20:20:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-18 20:20:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-18 20:20:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-18 20:20:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-18 20:20:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-18 20:20:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-18 20:20:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-18 20:20:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-18 20:20:50 WRN Prometheus returned 401. Re-authenticating... 2025-09-18 20:20:50 INF Re-authenticating Prometheus client 2025-09-18 20:20:50 WRN Prometheus returned 401. Re-authenticating... 2025-09-18 20:20:50 WRN Prometheus returned 401. Re-authenticating... 2025-09-18 20:20:50 WRN Prometheus returned 401. Re-authenticating... 2025-09-18 20:20:50 WRN Prometheus returned 401. Re-authenticating... 2025-09-18 20:20:50 WRN Prometheus returned 401. Re-authenticating... 2025-09-19 01:14:55 INF Adding event: Scenario_benchmark-stop 2025-09-19 01:38:26 INF Stopping Lodemon 2025-09-19 01:38:26 INF Stopping monitoring service...