==================================================================================================== ============================ Pod describe lodemon-6c5749b957-4mq8h ============================ ==================================================================================================== Name: lodemon-6c5749b957-4mq8h Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Tue, 16 Sep 2025 16:13:53 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6c5749b957 skaffold.dev/run-id=3a94d6aa-45f8-46ef-95d7-83fda26498b6 Annotations: Status: Running IP: 10.106.40.159 IPs: IP: 10.106.40.159 Controlled By: ReplicaSet/lodemon-6c5749b957 Containers: lodemon: Container ID: containerd://4c49b57f98927bbfd49facc2d015567943544519839a578d8fcac580ea98097b Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Tue, 16 Sep 2025 16:13:54 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-drbgf (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-drbgf: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-6c5749b957-4mq8h ============================== ==================================================================================================== 2025-09-16 16:13:54 INF Starting lodemon server on port 8080 2025-09-16 16:13:54 INF Initializing clients... 2025-09-16 16:13:54 DBG Fetching ID token for accessing prometheus 2025-09-16 16:13:54 DBG Fetched ID token successfully 2025-09-16 16:13:54 INF Clients initialized successfully 2025-09-16 16:14:30 INF Configuring Lodemon 2025-09-16 16:14:30 INF Configuring monitoring units 2025-09-16 16:14:30 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-16 16:14:30 TRC Parsing Prometheus queries from config 2025-09-16 16:14:30 TRC Parsing component query: DS Java heap mem usage 2025-09-16 16:14:30 TRC Parsing component query: IDM Java heap mem usage 2025-09-16 16:14:30 TRC Parsing component query: AM Java heap mem usage 2025-09-16 16:14:30 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-16 16:14:30 TRC Parsing global query: 2025-09-16 16:14:30 TRC Parsing scoped query: DS Identity backend connections 2025-09-16 16:14:30 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-16 16:14:30 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-16 16:14:30 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-16 16:14:30 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-16 16:14:30 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-16 16:14:30 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-16 16:14:30 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-16 16:14:30 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-16 16:14:30 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-16 16:14:30 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-16 16:14:30 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-16 16:14:30 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-16 16:14:30 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-16 16:14:30 INF  - All monitoring units configured 2025-09-16 16:14:30 INF Configuring monitoring service 2025-09-16 16:14:30 INF  - Adding monitoring units to the service 2025-09-16 16:14:30 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-16 16:14:30 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-16 16:14:30 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-16 16:14:30 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-16 16:14:30 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-16 16:14:30 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-16 16:14:30 INF  - All monitoring units added to the service 2025-09-16 16:14:30 INF Starting Lodemon now 2025-09-16 16:14:30 INF Starting monitoring service... 2025-09-16 16:14:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:14:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:14:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:14:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:14:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:14:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:14:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:14:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:14:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:14:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:14:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:14:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:14:34 INF Adding event: Scenario_benchmark 2025-09-16 16:14:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:14:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:14:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:14:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:14:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:14:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:14:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:14:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:14:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:14:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:14:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:14:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:14:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:14:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:14:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:14:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:14:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:14:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:14:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:14:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:14:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:14:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:14:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:14:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:14:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:14:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:15:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:15:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:15:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:15:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:15:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:15:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:15:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:15:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:15:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:15:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:15:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:15:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:15:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:15:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:15:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:15:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:15:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:15:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:15:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:15:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:15:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:15:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:15:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:15:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:15:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:15:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:15:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:15:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:15:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:15:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:15:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:15:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:15:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:15:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:15:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:15:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:15:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:15:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:15:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:15:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:15:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:15:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:15:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:15:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:15:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:15:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:15:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:15:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:15:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:15:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:15:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:15:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:15:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:15:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:15:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:15:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:15:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:15:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:15:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:15:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:15:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:15:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:15:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:15:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:15:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:15:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:15:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:15:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:16:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:16:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:16:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:16:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:16:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:16:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:16:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:16:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:16:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:16:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:16:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:16:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:16:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:16:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:16:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:16:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:16:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:16:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:16:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:16:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:16:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:16:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:16:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:16:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:16:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:16:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:16:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:16:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:16:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:16:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:16:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:16:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:16:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:16:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:16:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:16:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:16:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:16:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:16:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:16:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:16:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:16:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:16:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:16:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:16:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:16:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:16:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:16:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:16:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:16:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:16:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:16:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:16:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:16:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:16:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:16:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:16:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:16:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:16:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:16:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:16:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:16:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:16:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:16:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:16:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:16:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:16:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:16:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:16:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:17:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:17:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:17:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:17:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:17:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:17:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:17:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:17:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:17:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:17:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:17:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:17:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:17:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:17:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:17:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:17:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:17:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:17:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:17:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:17:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:17:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:17:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:17:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:17:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:17:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:17:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:17:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:17:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:17:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:17:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:17:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:17:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:17:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:17:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:17:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:17:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:17:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:17:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:17:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:17:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:17:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:17:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:17:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:17:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:17:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:17:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:17:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:17:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:17:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:17:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:17:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:17:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:17:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:17:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:17:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:17:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:17:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:17:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:17:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:17:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:17:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:17:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:17:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:17:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:17:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:17:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:17:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:17:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:17:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:17:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:17:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:17:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:17:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:17:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:18:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:18:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:18:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:18:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:18:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:18:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:18:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:18:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:18:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:18:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:18:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:18:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:18:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:18:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:18:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:18:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:18:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:18:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:18:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:18:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:18:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:18:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:18:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:18:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:18:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:18:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:18:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:18:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:18:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:18:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:18:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:18:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:18:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:18:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:18:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:18:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:18:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:18:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:18:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:18:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:18:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:18:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:18:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:18:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:18:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:18:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:18:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:18:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:18:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:18:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:18:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:18:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:18:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:18:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:18:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:18:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:18:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:18:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:18:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:18:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:18:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:18:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:18:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:18:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:18:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:18:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:18:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:18:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:18:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:18:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:18:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:18:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:18:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:19:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:19:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:19:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:19:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:19:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:19:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:19:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:19:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:19:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:19:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:19:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:19:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:19:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:19:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:19:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:19:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:19:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:19:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:19:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:19:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:19:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:19:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:19:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:19:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:19:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:19:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:19:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:19:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:19:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:19:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:19:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:19:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:19:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:19:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:19:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:19:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:19:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:19:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:19:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:19:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:19:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:19:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:19:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:19:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:19:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:19:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:19:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:19:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:19:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:19:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:19:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:19:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:19:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:19:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:19:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:19:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:19:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:19:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:19:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:19:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:19:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:19:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:19:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:19:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:19:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:19:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:19:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:19:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:19:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:20:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:20:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:20:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:20:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:20:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:20:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:20:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:20:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:20:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:20:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:20:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:20:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:20:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:20:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:20:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:20:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:20:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:20:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:20:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:20:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:20:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:20:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:20:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:20:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:20:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:20:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:20:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:20:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:20:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:20:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:20:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:20:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:20:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:20:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:20:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:20:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:20:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:20:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:20:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:20:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:20:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:20:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:20:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:20:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:20:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:20:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:20:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:20:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:20:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:20:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:20:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:20:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:20:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:20:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:20:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:20:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:20:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:20:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:20:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:20:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:20:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:20:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:20:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:20:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:20:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:20:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:20:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:20:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:20:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:20:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:20:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:20:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:20:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:20:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:21:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:21:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:21:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:21:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:21:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:21:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:21:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:21:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:21:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:21:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:21:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:21:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:21:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:21:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:21:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:21:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:21:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:21:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:21:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:21:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:21:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:21:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:21:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:21:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:21:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:21:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:21:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:21:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:21:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:21:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:21:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:21:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:21:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:21:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:21:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:21:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:21:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:21:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:21:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:21:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:21:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:21:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:21:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:21:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:21:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:21:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:21:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:21:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:21:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:21:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:21:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:21:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:21:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:21:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:21:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:21:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:21:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:21:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:21:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:21:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:21:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:21:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:21:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:21:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:21:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:21:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:21:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:21:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:21:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:21:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:21:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:21:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:21:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:22:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:22:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:22:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:22:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:22:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:22:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:22:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:22:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:22:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:22:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:22:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:22:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:22:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:22:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:22:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:22:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:22:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:22:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:22:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:22:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:22:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:22:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:22:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:22:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:22:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:22:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:22:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:22:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:22:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:22:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:22:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:22:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:22:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:22:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:22:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:22:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:22:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:22:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:22:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:22:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:22:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:22:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:22:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:22:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:22:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:22:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:22:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:22:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:22:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:22:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:22:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:22:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:22:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:22:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:22:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:22:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:22:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:22:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:22:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:22:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:22:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:22:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:22:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:22:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:22:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:22:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:22:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:22:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:22:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:23:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:23:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:23:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:23:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:23:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:23:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:23:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:23:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:23:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:23:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:23:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:23:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:23:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:23:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:23:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:23:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:23:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:23:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:23:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:23:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:23:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:23:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:23:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:23:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:23:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:23:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:23:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:23:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:23:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:23:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:23:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:23:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:23:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:23:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:23:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:23:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:23:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:23:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:23:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:23:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:23:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:23:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:23:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:23:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:23:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:23:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:23:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:23:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:23:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:23:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:23:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:23:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:23:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:23:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:23:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:23:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:23:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:23:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:23:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:23:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:23:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:23:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:23:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:23:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:23:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:23:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:23:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:23:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:23:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:23:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:23:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:23:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:23:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:24:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:24:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:24:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:24:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:24:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:24:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:24:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:24:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:24:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:24:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:24:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:24:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:24:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:24:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:24:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:24:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:24:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:24:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:24:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:24:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:24:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:24:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:24:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:24:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:24:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:24:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:24:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:24:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:24:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:24:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:24:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:24:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:24:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:24:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:24:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:24:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:24:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:24:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:24:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:24:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:24:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:24:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:24:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:24:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:24:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:24:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:24:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:24:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:24:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:24:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:24:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:24:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:24:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:24:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:24:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:24:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:24:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:24:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:24:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:24:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:24:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:24:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:24:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:24:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:24:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:24:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:24:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:24:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:25:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:25:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:25:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:25:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:25:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:25:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:25:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:25:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:25:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:25:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:25:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:25:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:25:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:25:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:25:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:25:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:25:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:25:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:25:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:25:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:25:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:25:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:25:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:25:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:25:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:25:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:25:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:25:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:25:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:25:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:25:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:25:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:25:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:25:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:25:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:25:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:25:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:25:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:25:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:25:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:25:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:25:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:25:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:25:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:25:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:25:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:25:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:25:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:25:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:25:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:25:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:25:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:25:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:25:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:25:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:25:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:25:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:25:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:25:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:25:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:25:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:25:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:25:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:25:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:25:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:25:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:25:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:25:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:26:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:26:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:26:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:26:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:26:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:26:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:26:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:26:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:26:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:26:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:26:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:26:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:26:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:26:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:26:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:26:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:26:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:26:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:26:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:26:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:26:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:26:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:26:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:26:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:26:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:26:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:26:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:26:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:26:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:26:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:26:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:26:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:26:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:26:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:26:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:26:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:26:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:26:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:26:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:26:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:26:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:26:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:26:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:26:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:26:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:26:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:26:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:26:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:26:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:26:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:26:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:26:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:26:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:26:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:26:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:26:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:26:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:26:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:26:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:26:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:26:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:26:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:26:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:26:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:26:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:26:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:26:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:26:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:27:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:27:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:27:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:27:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:27:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:27:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:27:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:27:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:27:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:27:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:27:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:27:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:27:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:27:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:27:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:27:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:27:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:27:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:27:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:27:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:27:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:27:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:27:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:27:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:27:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:27:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:27:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:27:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:27:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:27:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:27:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:27:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:27:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:27:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:27:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:27:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:27:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:27:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:27:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:27:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:27:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:27:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:27:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:27:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:27:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:27:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:27:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:27:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:27:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:27:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:27:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:27:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:27:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:27:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:27:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:27:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:27:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:27:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:27:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:27:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:27:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:27:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:27:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:27:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:27:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:27:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:27:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:27:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:27:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:27:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:27:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:27:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:27:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:27:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:28:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:28:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:28:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:28:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:28:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:28:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:28:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:28:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:28:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:28:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:28:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:28:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:28:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:28:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:28:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:28:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:28:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:28:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:28:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:28:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:28:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:28:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:28:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:28:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:28:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:28:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:28:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:28:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:28:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:28:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:28:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:28:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:28:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:28:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:28:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:28:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:28:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:28:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:28:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:28:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:28:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:28:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:28:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:28:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:28:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:28:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:28:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:28:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:28:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:28:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:28:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:28:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:28:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:28:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:28:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:28:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:28:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:28:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:28:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:28:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:28:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:28:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:28:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:28:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:28:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:28:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:28:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:28:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:29:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:29:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:29:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:29:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:29:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:29:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:29:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:29:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:29:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:29:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:29:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:29:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:29:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:29:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:29:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:29:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:29:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:29:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:29:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:29:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:29:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:29:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:29:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:29:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:29:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:29:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:29:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:29:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:29:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:29:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:29:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:29:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:29:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:29:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:29:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:29:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:29:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:29:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:29:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:29:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:29:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:29:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:29:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:29:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:29:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:29:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:29:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:29:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:29:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:29:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:29:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:29:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:29:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:29:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:29:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:29:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:29:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:29:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:29:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:29:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:29:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:29:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:29:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:29:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:29:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:29:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:29:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:29:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:30:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:30:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:30:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:30:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:30:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:30:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:30:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:30:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:30:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:30:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:30:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:30:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:30:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:30:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:30:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:30:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:30:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:30:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:30:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:30:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:30:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:30:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:30:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:30:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:30:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:30:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:30:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:30:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:30:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:30:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:30:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:30:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:30:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:30:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:30:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:30:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:30:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:30:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:30:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:30:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:30:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:30:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:30:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:30:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:30:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:30:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:30:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:30:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:30:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:30:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:30:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:30:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:30:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:30:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:30:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:30:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:30:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:30:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:30:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:30:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:30:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:30:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:30:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:30:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:30:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:30:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:30:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:30:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:31:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:31:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:31:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:31:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:31:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:31:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:31:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:31:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:31:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:31:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:31:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:31:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:31:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:31:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:31:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:31:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:31:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:31:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:31:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:31:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:31:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:31:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:31:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:31:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:31:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:31:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:31:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:31:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:31:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:31:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:31:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:31:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:31:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:31:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:31:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:31:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:31:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:31:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:31:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:31:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:31:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:31:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:31:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:31:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:31:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:31:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:31:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:31:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:31:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:31:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:31:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:31:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:31:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:31:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:31:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:31:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:31:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:31:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:31:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:31:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:31:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:31:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:31:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:31:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:31:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:31:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:31:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:31:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:32:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:32:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:32:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:32:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:32:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:32:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:32:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:32:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:32:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:32:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:32:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:32:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:32:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:32:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:32:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:32:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:32:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:32:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:32:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:32:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:32:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:32:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:32:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:32:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:32:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:32:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:32:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:32:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:32:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:32:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:32:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:32:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:32:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:32:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:32:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:32:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:32:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:32:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:32:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:32:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:32:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:32:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:32:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:32:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:32:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:32:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:32:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:32:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:32:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:32:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:32:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:32:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:32:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:32:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:32:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:32:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:32:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:32:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:32:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:32:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:32:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:32:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:32:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:33:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:33:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:33:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:33:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:33:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:33:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:33:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:33:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:33:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:33:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:33:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:33:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:33:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:33:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:33:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:33:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:33:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:33:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:33:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:33:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:33:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:33:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:33:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:33:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:33:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:33:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:33:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:33:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:33:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:33:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:33:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:33:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:33:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:33:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:33:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:33:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:33:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:33:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:33:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:33:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:33:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:33:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:33:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:33:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:33:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:33:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:33:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:33:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:33:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:33:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:33:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:33:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:33:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:33:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:33:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:33:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:33:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:33:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:33:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:33:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:33:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:33:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:33:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:33:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:33:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:33:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:33:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:33:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:33:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:33:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:33:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:33:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:33:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:33:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:34:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:34:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:34:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:34:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:34:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:34:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:34:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:34:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:34:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:34:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:34:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:34:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:34:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:34:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:34:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:34:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:34:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:34:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:34:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:34:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:34:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:34:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:34:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:34:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:34:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:34:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:34:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:34:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:34:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:34:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:34:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:34:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:34:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:34:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:34:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:34:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:34:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:34:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:34:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:34:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:34:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:34:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:34:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:34:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:34:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:34:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:34:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:34:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:34:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:34:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:34:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:34:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:34:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:34:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:34:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:34:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:34:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:34:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:34:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:34:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:34:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:34:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:34:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:34:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:34:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:34:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:34:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:34:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:34:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:34:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:34:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:34:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:34:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:35:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:35:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:35:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:35:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:35:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:35:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:35:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:35:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:35:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:35:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:35:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:35:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:35:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:35:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:35:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:35:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:35:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:35:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:35:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:35:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:35:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:35:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:35:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:35:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:35:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:35:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:35:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:35:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:35:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:35:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:35:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:35:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:35:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:35:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:35:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:35:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:35:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:35:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:35:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:35:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:35:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:35:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:35:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:35:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:35:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:35:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:35:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:35:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:35:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:35:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:35:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:35:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:35:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:35:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:35:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:35:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:35:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:35:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:35:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:35:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:35:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:35:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:35:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:35:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:35:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:35:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:35:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:35:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:35:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:36:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:36:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:36:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:36:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:36:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:36:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:36:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:36:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:36:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:36:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:36:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:36:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:36:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:36:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:36:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:36:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:36:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:36:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:36:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:36:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:36:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:36:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:36:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:36:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:36:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:36:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:36:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:36:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:36:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:36:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:36:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:36:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:36:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:36:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:36:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:36:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:36:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:36:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:36:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:36:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:36:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:36:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:36:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:36:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:36:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:36:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:36:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:36:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:36:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:36:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:36:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:36:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:36:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:36:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:36:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:36:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:36:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:36:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:36:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:36:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:36:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:36:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:36:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:36:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:36:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:36:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:36:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:36:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:36:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:36:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:36:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:36:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:36:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:36:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:37:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:37:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:37:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:37:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:37:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:37:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:37:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:37:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:37:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:37:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:37:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:37:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:37:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:37:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:37:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:37:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:37:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:37:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:37:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:37:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:37:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:37:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:37:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:37:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:37:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:37:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:37:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:37:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:37:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:37:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:37:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:37:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:37:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:37:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:37:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:37:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:37:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:37:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:37:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:37:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:37:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:37:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:37:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:37:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:37:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:37:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:37:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:37:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:37:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:37:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:37:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:37:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:37:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:37:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:37:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:37:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:37:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:37:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:37:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:37:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:37:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:37:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:37:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:37:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:37:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:37:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:37:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:37:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:37:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:37:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:37:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:37:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:37:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:37:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:38:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:38:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:38:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:38:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:38:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:38:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:38:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:38:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:38:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:38:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:38:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:38:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:38:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:38:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:38:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:38:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:38:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:38:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:38:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:38:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:38:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:38:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:38:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:38:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:38:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:38:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:38:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:38:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:38:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:38:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:38:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:38:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:38:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:38:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:38:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:38:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:38:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:38:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:38:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:38:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:38:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:38:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:38:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:38:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:38:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:38:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:38:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:38:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:38:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:38:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:38:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:38:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:38:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:38:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:38:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:38:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:38:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:38:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:38:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:38:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:38:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:38:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:38:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:38:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:38:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:38:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:38:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:38:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:38:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:38:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:38:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:38:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:38:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:38:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:39:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:39:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:39:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:39:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:39:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:39:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:39:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:39:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:39:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:39:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:39:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:39:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:39:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:39:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:39:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:39:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:39:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:39:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:39:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:39:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:39:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:39:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:39:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:39:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:39:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:39:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:39:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:39:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:39:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:39:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:39:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:39:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:39:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:39:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:39:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:39:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:39:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:39:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:39:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:39:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:39:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:39:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:39:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:39:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:39:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:39:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:39:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:39:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:39:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:39:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:39:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:39:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:39:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:39:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:39:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:39:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:39:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:39:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:39:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:39:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:39:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:39:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:39:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:39:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:39:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:39:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:39:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:39:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:39:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:39:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:39:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:39:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:39:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:40:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:40:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:40:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:40:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:40:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:40:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:40:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:40:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:40:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:40:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:40:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:40:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:40:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:40:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:40:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:40:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:40:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:40:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:40:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:40:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:40:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:40:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:40:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:40:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:40:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:40:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:40:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:40:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:40:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:40:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:40:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:40:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:40:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:40:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:40:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:40:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:40:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:40:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:40:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:40:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:40:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:40:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:40:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:40:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:40:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:40:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:40:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:40:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:40:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:40:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:40:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:40:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:40:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:40:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:40:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:40:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:40:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:40:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:40:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:40:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:40:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:40:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:40:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:40:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:40:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:40:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:40:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:40:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:41:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:41:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:41:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:41:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:41:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:41:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:41:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:41:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:41:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:41:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:41:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:41:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:41:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:41:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:41:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:41:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:41:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:41:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:41:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:41:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:41:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:41:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:41:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:41:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:41:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:41:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:41:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:41:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:41:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:41:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:41:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:41:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:41:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:41:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:41:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:41:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:41:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:41:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:41:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:41:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:41:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:41:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:41:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:41:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:41:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:41:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:41:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:41:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:41:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:41:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:41:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:41:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:41:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:41:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:41:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:41:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:41:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:41:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:41:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:41:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:41:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:41:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:41:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:41:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:41:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:41:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:41:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:41:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:41:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:41:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:41:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:41:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:41:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:41:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:42:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:42:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:42:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:42:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:42:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:42:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:42:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:42:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:42:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:42:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:42:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:42:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:42:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:42:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:42:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:42:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:42:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:42:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:42:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:42:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:42:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:42:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:42:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:42:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:42:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:42:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:42:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:42:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:42:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:42:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:42:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:42:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:42:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:42:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:42:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:42:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:42:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:42:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:42:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:42:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:42:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:42:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:42:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:42:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:42:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:42:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:42:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:42:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:42:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:42:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:42:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:42:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:42:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:42:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:42:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:42:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:42:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:42:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:42:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:42:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:42:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:42:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:42:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:42:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:42:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:42:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:42:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:42:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:42:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:43:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:43:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:43:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:43:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:43:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:43:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:43:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:43:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:43:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:43:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:43:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:43:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:43:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:43:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:43:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:43:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:43:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:43:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:43:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:43:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:43:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:43:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:43:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:43:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:43:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:43:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:43:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:43:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:43:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:43:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:43:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:43:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:43:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:43:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:43:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:43:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:43:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:43:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:43:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:43:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:43:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:43:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:43:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:43:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:43:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:43:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:43:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:43:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:43:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:43:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:43:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:43:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:43:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:43:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:43:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:43:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:43:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:43:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:43:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:43:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:43:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:43:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:43:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:43:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:43:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:43:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:43:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:43:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:43:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:43:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:43:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:43:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:43:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:43:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:44:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:44:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:44:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:44:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:44:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:44:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:44:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:44:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:44:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:44:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:44:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:44:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:44:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:44:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:44:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:44:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:44:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:44:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:44:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:44:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:44:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:44:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:44:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:44:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:44:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:44:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:44:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:44:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:44:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:44:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:44:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:44:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:44:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:44:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:44:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:44:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:44:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:44:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:44:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:44:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:44:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:44:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:44:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:44:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:44:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:44:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:44:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:44:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:44:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:44:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:44:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:44:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:44:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:44:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:44:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:44:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:44:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:44:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:44:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:44:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:44:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:44:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:44:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:44:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:44:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:44:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:44:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:44:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:44:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:44:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:44:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:44:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:44:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:45:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:45:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:45:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:45:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:45:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:45:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:45:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:45:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:45:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:45:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:45:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:45:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:45:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:45:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:45:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:45:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:45:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:45:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:45:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:45:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:45:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:45:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:45:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:45:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:45:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:45:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:45:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:45:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:45:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:45:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:45:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:45:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:45:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:45:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:45:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:45:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:45:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:45:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:45:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:45:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:45:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:45:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:45:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:45:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:45:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:45:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:45:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:45:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:45:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:45:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:45:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:45:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:45:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:45:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:45:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:45:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:45:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:45:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:45:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:45:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:45:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:45:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:45:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:45:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:45:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:45:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:45:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:45:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:45:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:46:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:46:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:46:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:46:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:46:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:46:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:46:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:46:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:46:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:46:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:46:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:46:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:46:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:46:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:46:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:46:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:46:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:46:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:46:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:46:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:46:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:46:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:46:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:46:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:46:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:46:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:46:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:46:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:46:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:46:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:46:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:46:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:46:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:46:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:46:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:46:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:46:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:46:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:46:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:46:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:46:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:46:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:46:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:46:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:46:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:46:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:46:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:46:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:46:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:46:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:46:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:46:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:46:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:46:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:46:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:46:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:46:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:46:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:46:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:46:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:46:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:46:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:46:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:46:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:46:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:46:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:46:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:46:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:46:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:46:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:46:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:46:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:46:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:46:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:47:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:47:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:47:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:47:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:47:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:47:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:47:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:47:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:47:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:47:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:47:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:47:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:47:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:47:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:47:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:47:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:47:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:47:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:47:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:47:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:47:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:47:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:47:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:47:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:47:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:47:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:47:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:47:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:47:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:47:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:47:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:47:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:47:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:47:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:47:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:47:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:47:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:47:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:47:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:47:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:47:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:47:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:47:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:47:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:47:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:47:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:47:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:47:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:47:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:47:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:47:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:47:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:47:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:47:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:47:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:47:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:47:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:47:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:47:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:47:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:47:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:47:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:47:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:47:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:47:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:47:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:47:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:47:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:47:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:47:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:47:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:47:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:47:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:47:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:48:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:48:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:48:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:48:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:48:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:48:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:48:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:48:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:48:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:48:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:48:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:48:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:48:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:48:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:48:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:48:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:48:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:48:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:48:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:48:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:48:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:48:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:48:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:48:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:48:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:48:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:48:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:48:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:48:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:48:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:48:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:48:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:48:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:48:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:48:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:48:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:48:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:48:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:48:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:48:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:48:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:48:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:48:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:48:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:48:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:48:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:48:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:48:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:48:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:48:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:48:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:48:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:48:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:48:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:48:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:48:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:48:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:48:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:48:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:48:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:48:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:48:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:48:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:48:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:48:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:48:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:48:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:48:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:48:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:48:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:48:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:48:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:48:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:49:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:49:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:49:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:49:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:49:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:49:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:49:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:49:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:49:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:49:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:49:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:49:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:49:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:49:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:49:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:49:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:49:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:49:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:49:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:49:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:49:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:49:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:49:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:49:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:49:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:49:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:49:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:49:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:49:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:49:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:49:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:49:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:49:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:49:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:49:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:49:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:49:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:49:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:49:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:49:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:49:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:49:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:49:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:49:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:49:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:49:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:49:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:49:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:49:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:49:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:49:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:49:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:49:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:49:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:49:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:49:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:49:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:49:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:49:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:49:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:49:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:49:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:49:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:49:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:49:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:49:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:49:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:49:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:49:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:49:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:49:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:49:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:49:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:49:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:50:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:50:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:50:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:50:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:50:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:50:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:50:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:50:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:50:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:50:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:50:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:50:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:50:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:50:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:50:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:50:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:50:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:50:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:50:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:50:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:50:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:50:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:50:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:50:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:50:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:50:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:50:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:50:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:50:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:50:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:50:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:50:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:50:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:50:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:50:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:50:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:50:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:50:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:50:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:50:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:50:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:50:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:50:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:50:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:50:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:50:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:50:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:50:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:50:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:50:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:50:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:50:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:50:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:50:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:50:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:50:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:50:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:50:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:50:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:50:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:50:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:50:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:51:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:51:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:51:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:51:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:51:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:51:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:51:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:51:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:51:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:51:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:51:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:51:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:51:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:51:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:51:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:51:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:51:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:51:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:51:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:51:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:51:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:51:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:51:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:51:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:51:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:51:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:51:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:51:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:51:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:51:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:51:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:51:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:51:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:51:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:51:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:51:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:51:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:51:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:51:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:51:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:51:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:51:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:51:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:51:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:51:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:51:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:51:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:51:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:51:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:51:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:51:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:51:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:51:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:51:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:51:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:51:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:51:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:51:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:51:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:51:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:51:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:51:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:51:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:51:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:51:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:51:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:51:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:51:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:51:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:51:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:51:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:51:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:51:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:51:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:52:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:52:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:52:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:52:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:52:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:52:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:52:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:52:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:52:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:52:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:52:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:52:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:52:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:52:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:52:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:52:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:52:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:52:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:52:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:52:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:52:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:52:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:52:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:52:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:52:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:52:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:52:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:52:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:52:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:52:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:52:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:52:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:52:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:52:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:52:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:52:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:52:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:52:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:52:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:52:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:52:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:52:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:52:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:52:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:52:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:52:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:52:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:52:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:52:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:52:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:52:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:52:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:52:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:52:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:52:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:52:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:52:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:52:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:52:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:52:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:52:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:52:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:52:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:52:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:52:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:52:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:52:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:52:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:53:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:53:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:53:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:53:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:53:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:53:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:53:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:53:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:53:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:53:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:53:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:53:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:53:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:53:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:53:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:53:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:53:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:53:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:53:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:53:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:53:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:53:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:53:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:53:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:53:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:53:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:53:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:53:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:53:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:53:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:53:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:53:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:53:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:53:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:53:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:53:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:53:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:53:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:53:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:53:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:53:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:53:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:53:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:53:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:53:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:53:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:53:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:53:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:53:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:53:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:53:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:53:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:53:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:53:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:53:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:53:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:53:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:53:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:53:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:53:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:53:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:53:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:53:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:53:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:53:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:53:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:53:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:53:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:54:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:54:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:54:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:54:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:54:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:54:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:54:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:54:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:54:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:54:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:54:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:54:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:54:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:54:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:54:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:54:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:54:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:54:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:54:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:54:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:54:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:54:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:54:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:54:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:54:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:54:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:54:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:54:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:54:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:54:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:54:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:54:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:54:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:54:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:54:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:54:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:54:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:54:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:54:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:54:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:54:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:54:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:54:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:54:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:54:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:54:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:54:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:54:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:54:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:54:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:54:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:54:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:54:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:54:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:54:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:54:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:54:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:54:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:54:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:54:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:54:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:54:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:54:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:54:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:54:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:54:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:54:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:54:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:55:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:55:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:55:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:55:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:55:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:55:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:55:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:55:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:55:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:55:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:55:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:55:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:55:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:55:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:55:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:55:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:55:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:55:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:55:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:55:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:55:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:55:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:55:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:55:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:55:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:55:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:55:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:55:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:55:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:55:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:55:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:55:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:55:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:55:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:55:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:55:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:55:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:55:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:55:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:55:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:55:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:55:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:55:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:55:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:55:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:55:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:55:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:55:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:55:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:55:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:55:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:55:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:55:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:55:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:55:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:55:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:55:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:55:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:55:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:55:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:55:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:55:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:55:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:55:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:55:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:55:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:55:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:55:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:56:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:56:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:56:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:56:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:56:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:56:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:56:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:56:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:56:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:56:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:56:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:56:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:56:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:56:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:56:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:56:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:56:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:56:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:56:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:56:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:56:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:56:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:56:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:56:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:56:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:56:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:56:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:56:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:56:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:56:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:56:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:56:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:56:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:56:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:56:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:56:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:56:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:56:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:56:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:56:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:56:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:56:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:56:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:56:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:56:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:56:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:56:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:56:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:56:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:56:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:56:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:56:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:56:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:56:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:56:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:56:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:56:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:56:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:56:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:56:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:56:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:56:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:56:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:56:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:56:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:56:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:56:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:56:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:57:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:57:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:57:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:57:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:57:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:57:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:57:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:57:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:57:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:57:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:57:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:57:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:57:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:57:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:57:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:57:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:57:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:57:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:57:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:57:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:57:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:57:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:57:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:57:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:57:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:57:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:57:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:57:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:57:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:57:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:57:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:57:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:57:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:57:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:57:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:57:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:57:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:57:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:57:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:57:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:57:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:57:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:57:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:57:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:57:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:57:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:57:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:57:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:57:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:57:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:57:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:57:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:57:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:57:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:57:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:57:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:57:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:57:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:57:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:57:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:57:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:57:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:57:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:57:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:57:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:57:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:57:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:57:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:57:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:57:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:57:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:57:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:57:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:57:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:58:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:58:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:58:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:58:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:58:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:58:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:58:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:58:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:58:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:58:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:58:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:58:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:58:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:58:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:58:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:58:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:58:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:58:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:58:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:58:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:58:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:58:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:58:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:58:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:58:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:58:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:58:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:58:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:58:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:58:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:58:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:58:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:58:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:58:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:58:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:58:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:58:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:58:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:58:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:58:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:58:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:58:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:58:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:58:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:58:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:58:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:58:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:58:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:58:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:58:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:58:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:58:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:58:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:58:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:58:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:58:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:58:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:58:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:58:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:58:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:58:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:58:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:58:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:58:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:58:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:58:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:58:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:58:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:58:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:59:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:59:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:59:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:59:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:59:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:59:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:59:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:59:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:59:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:59:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:59:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:59:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:59:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:59:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:59:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:59:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:59:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:59:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:59:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:59:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:59:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:59:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:59:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:59:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:59:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:59:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:59:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:59:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:59:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:59:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:59:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:59:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:59:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:59:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:59:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:59:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:59:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:59:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:59:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:59:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:59:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:59:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:59:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:59:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:59:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:59:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:59:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:59:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:59:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:59:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:59:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:59:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:59:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:59:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:59:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:59:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:59:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:59:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:59:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 16:59:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 16:59:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 16:59:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 16:59:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 16:59:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 16:59:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 16:59:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 16:59:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 16:59:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 16:59:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 16:59:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 16:59:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 16:59:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 16:59:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:00:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:00:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:00:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:00:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:00:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:00:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:00:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:00:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:00:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:00:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:00:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:00:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:00:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:00:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:00:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:00:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:00:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:00:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:00:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:00:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:00:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:00:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:00:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:00:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:00:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:00:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:00:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:00:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:00:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:00:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:00:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:00:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:00:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:00:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:00:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:00:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:00:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:00:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:00:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:00:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:00:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:00:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:00:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:00:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:00:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:00:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:00:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:00:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:00:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:00:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:00:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:00:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:00:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:00:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:00:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:00:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:00:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:00:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:00:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:00:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:00:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:00:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:00:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:00:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:00:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:00:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:00:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:00:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:01:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:01:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:01:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:01:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:01:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:01:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:01:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:01:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:01:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:01:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:01:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:01:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:01:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:01:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:01:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:01:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:01:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:01:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:01:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:01:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:01:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:01:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:01:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:01:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:01:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:01:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:01:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:01:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:01:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:01:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:01:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:01:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:01:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:01:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:01:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:01:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:01:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:01:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:01:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:01:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:01:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:01:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:01:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:01:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:01:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:01:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:01:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:01:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:01:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:01:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:01:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:01:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:01:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:01:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:01:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:01:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:01:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:01:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:01:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:01:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:01:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:01:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:01:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:01:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:01:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:01:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:01:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:01:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:02:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:02:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:02:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:02:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:02:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:02:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:02:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:02:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:02:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:02:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:02:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:02:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:02:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:02:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:02:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:02:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:02:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:02:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:02:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:02:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:02:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:02:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:02:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:02:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:02:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:02:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:02:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:02:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:02:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:02:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:02:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:02:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:02:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:02:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:02:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:02:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:02:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:02:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:02:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:02:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:02:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:02:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:02:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:02:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:02:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:02:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:02:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:02:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:02:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:02:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:02:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:02:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:02:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:02:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:02:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:02:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:02:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:02:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:02:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:02:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:02:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:02:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:02:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:02:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:02:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:02:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:02:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:02:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:02:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:03:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:03:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:03:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:03:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:03:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:03:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:03:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:03:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:03:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:03:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:03:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:03:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:03:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:03:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:03:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:03:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:03:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:03:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:03:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:03:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:03:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:03:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:03:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:03:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:03:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:03:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:03:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:03:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:03:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:03:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:03:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:03:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:03:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:03:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:03:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:03:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:03:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:03:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:03:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:03:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:03:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:03:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:03:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:03:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:03:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:03:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:03:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:03:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:03:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:03:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:03:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:03:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:03:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:03:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:03:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:03:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:03:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:03:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:03:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:03:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:03:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:03:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:03:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:03:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:03:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:03:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:03:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:03:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:03:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:03:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:03:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:03:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:03:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:04:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:04:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:04:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:04:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:04:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:04:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:04:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:04:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:04:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:04:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:04:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:04:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:04:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:04:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:04:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:04:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:04:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:04:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:04:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:04:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:04:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:04:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:04:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:04:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:04:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:04:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:04:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:04:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:04:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:04:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:04:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:04:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:04:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:04:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:04:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:04:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:04:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:04:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:04:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:04:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:04:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:04:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:04:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:04:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:04:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:04:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:04:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:04:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:04:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:04:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:04:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:04:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:04:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:04:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:04:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:04:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:04:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:04:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:04:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:04:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:04:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:04:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:04:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:04:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:04:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:04:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:04:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:04:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:05:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:05:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:05:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:05:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:05:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:05:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:05:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:05:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:05:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:05:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:05:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:05:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:05:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:05:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:05:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:05:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:05:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:05:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:05:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:05:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:05:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:05:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:05:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:05:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:05:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:05:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:05:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:05:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:05:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:05:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:05:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:05:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:05:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:05:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:05:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:05:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:05:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:05:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:05:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:05:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:05:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:05:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:05:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:05:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:05:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:05:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:05:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:05:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:05:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:05:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:05:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:05:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:05:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:05:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:05:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:05:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:05:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:05:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:05:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:05:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:05:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:05:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:05:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:05:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:05:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:05:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:05:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:05:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:05:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:05:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:05:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:05:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:05:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:05:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:06:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:06:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:06:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:06:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:06:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:06:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:06:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:06:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:06:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:06:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:06:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:06:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:06:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:06:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:06:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:06:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:06:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:06:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:06:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:06:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:06:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:06:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:06:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:06:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:06:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:06:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:06:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:06:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:06:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:06:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:06:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:06:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:06:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:06:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:06:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:06:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:06:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:06:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:06:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:06:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:06:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:06:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:06:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:06:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:06:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:06:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:06:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:06:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:06:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:06:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:06:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:06:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:06:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:06:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:06:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:06:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:06:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:06:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:06:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:06:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:06:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:06:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:06:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:06:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:06:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:06:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:06:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:06:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:06:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:07:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:07:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:07:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:07:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:07:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:07:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:07:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:07:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:07:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:07:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:07:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:07:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:07:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:07:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:07:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:07:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:07:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:07:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:07:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:07:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:07:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:07:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:07:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:07:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:07:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:07:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:07:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:07:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:07:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:07:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:07:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:07:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:07:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:07:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:07:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:07:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:07:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:07:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:07:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:07:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:07:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:07:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:07:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:07:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:07:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:07:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:07:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:07:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:07:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:07:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:07:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:07:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:07:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:07:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:07:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:07:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:07:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:07:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:07:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:07:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:07:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:07:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:07:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:07:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:07:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:07:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:07:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:07:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:07:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:07:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:07:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:07:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:07:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:08:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:08:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:08:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:08:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:08:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:08:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:08:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:08:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:08:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:08:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:08:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:08:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:08:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:08:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:08:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:08:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:08:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:08:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:08:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:08:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:08:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:08:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:08:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:08:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:08:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:08:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:08:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:08:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:08:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:08:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:08:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:08:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:08:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:08:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:08:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:08:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:08:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:08:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:08:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:08:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:08:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:08:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:08:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:08:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:08:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:08:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:08:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:08:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:08:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:08:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:08:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:08:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:08:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:08:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:08:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:08:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:08:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:08:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:08:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:08:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:08:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:08:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:08:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:08:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:08:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:08:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:08:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:08:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:09:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:09:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:09:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:09:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:09:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:09:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:09:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:09:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:09:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:09:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:09:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:09:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:09:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:09:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:09:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:09:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:09:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:09:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:09:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:09:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:09:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:09:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:09:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:09:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:09:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:09:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:09:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:09:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:09:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:09:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:09:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:09:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:09:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:09:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:09:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:09:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:09:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:09:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:09:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:09:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:09:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:09:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:09:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:09:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:09:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:09:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:09:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:09:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:09:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:09:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:09:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:09:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:09:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:09:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:09:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:09:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:09:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:09:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:09:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:09:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:09:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:09:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:09:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:10:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:10:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:10:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:10:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:10:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:10:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:10:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:10:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:10:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:10:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:10:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:10:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:10:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:10:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:10:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:10:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:10:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:10:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:10:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:10:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:10:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:10:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:10:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:10:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:10:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:10:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:10:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:10:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:10:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:10:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:10:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:10:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:10:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:10:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:10:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:10:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:10:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:10:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:10:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:10:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:10:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:10:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:10:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:10:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:10:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:10:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:10:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:10:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:10:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:10:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:10:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:10:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:10:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:10:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:10:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:10:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:10:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:10:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:10:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:10:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:10:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:10:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:10:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:10:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:10:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:10:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:10:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:10:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:10:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:10:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:10:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:10:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:10:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:11:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:11:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:11:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:11:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:11:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:11:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:11:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:11:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:11:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:11:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:11:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:11:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:11:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:11:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:11:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:11:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:11:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:11:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:11:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:11:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:11:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:11:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:11:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:11:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:11:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:11:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:11:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:11:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:11:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:11:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:11:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:11:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:11:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:11:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:11:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:11:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:11:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:11:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:11:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:11:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:11:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:11:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:11:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:11:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:11:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:11:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:11:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:11:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:11:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:11:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:11:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:11:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:11:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:11:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:11:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:11:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:11:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:11:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:11:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:11:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:11:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:11:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:11:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:11:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:11:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:11:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:11:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:11:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:11:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:12:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:12:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:12:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:12:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:12:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:12:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:12:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:12:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:12:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:12:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:12:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:12:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:12:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:12:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:12:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:12:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:12:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:12:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:12:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:12:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:12:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:12:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:12:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:12:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:12:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:12:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:12:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:12:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:12:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:12:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:12:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:12:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:12:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:12:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:12:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:12:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:12:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:12:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:12:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:12:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:12:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:12:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:12:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:12:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:12:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:12:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:12:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:12:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:12:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:12:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:12:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:12:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:12:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:12:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:12:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:12:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:12:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:12:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:12:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:12:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:12:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:12:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:12:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:12:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:12:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:12:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:12:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:12:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:12:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:12:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:12:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:12:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:12:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:12:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:13:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:13:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:13:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:13:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:13:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:13:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:13:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:13:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:13:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:13:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:13:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:13:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:13:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:13:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:13:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:13:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:13:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:13:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:13:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:13:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:13:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:13:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:13:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:13:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:13:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:13:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:13:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:13:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:13:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:13:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:13:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:13:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:13:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:13:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:13:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:13:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:13:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:13:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:13:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:13:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:13:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:13:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:13:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:13:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:13:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:13:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:13:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:13:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:13:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:13:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:13:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:13:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:13:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:13:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:13:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:13:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:13:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:13:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:13:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:13:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:13:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:13:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:13:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:13:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:13:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:13:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:13:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:13:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:13:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:13:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:13:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:13:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:13:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:13:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:14:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:14:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:14:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:14:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:14:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:14:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:14:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:14:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:14:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:14:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:14:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:14:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:14:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:14:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:14:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:14:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:14:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:14:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:14:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:14:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:14:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:14:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:14:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:14:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:14:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:14:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:14:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:14:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:14:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:14:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:14:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:14:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:14:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:14:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:14:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:14:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:14:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:14:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:14:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:14:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:14:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:14:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:14:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:14:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:14:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:14:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:14:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:14:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:14:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:14:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:14:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:14:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:14:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:14:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:14:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:14:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:14:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:14:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:14:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:14:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:14:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:14:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:14:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:14:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:14:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:14:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:14:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:14:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:14:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:14:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:14:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:14:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:14:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:14:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:15:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:15:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:15:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:15:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:15:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:15:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:15:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:15:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:15:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:15:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:15:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:15:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:15:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:15:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:15:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:15:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:15:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:15:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:15:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:15:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:15:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:15:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:15:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:15:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:15:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:15:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:15:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:15:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:15:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:15:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:15:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:15:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:15:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:15:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:15:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:15:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:15:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:15:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:15:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:15:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:15:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:15:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:15:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:15:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:15:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:15:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:15:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:15:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:15:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:15:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:15:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:15:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:15:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:15:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:15:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:15:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:15:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:15:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:15:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:15:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:15:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:15:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:15:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:15:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:15:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:15:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:15:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:15:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:15:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:15:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:15:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:15:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:15:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:15:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:16:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:16:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:16:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:16:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:16:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:16:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:16:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:16:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:16:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:16:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:16:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:16:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:16:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:16:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:16:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:16:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:16:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:16:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:16:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:16:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:16:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:16:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:16:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:16:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:16:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:16:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:16:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:16:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:16:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:16:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:16:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:16:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:16:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:16:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:16:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:16:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:16:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:16:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:16:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:16:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:16:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:16:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:16:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:16:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:16:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:16:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:16:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:16:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:16:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:16:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:16:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:16:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:16:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:16:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:16:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:16:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:16:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:16:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:16:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:16:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:16:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:16:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:16:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:16:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:16:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:16:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:16:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:16:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:16:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:16:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:16:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:16:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:16:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:17:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:17:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:17:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:17:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:17:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:17:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:17:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:17:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:17:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:17:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:17:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:17:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:17:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:17:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:17:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:17:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:17:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:17:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:17:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:17:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:17:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:17:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:17:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:17:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:17:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:17:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:17:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:17:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:17:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:17:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:17:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:17:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:17:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:17:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:17:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:17:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:17:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:17:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:17:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:17:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:17:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:17:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:17:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:17:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:17:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:17:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:17:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:17:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:17:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:17:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:17:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:17:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:17:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:17:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:17:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:17:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:17:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:17:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:17:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:17:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:17:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:17:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:17:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:17:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:17:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:17:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:17:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:17:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:17:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:18:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:18:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:18:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:18:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:18:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:18:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:18:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:18:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:18:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:18:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:18:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:18:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:18:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:18:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:18:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:18:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:18:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:18:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:18:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:18:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:18:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:18:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:18:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:18:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:18:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:18:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:18:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:18:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:18:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:18:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:18:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:18:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:18:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:18:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:18:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:18:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:18:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:18:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:18:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:18:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:18:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:18:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:18:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:18:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:18:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:18:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:18:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:18:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:18:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:18:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:18:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:18:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:18:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:18:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:18:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:18:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:18:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 17:18:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 17:18:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 17:18:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 17:18:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 17:18:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 17:18:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 17:18:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 17:18:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 17:18:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 17:18:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 17:18:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 17:18:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 17:18:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 17:18:55 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 17:18:55 INF Re-authenticating Prometheus client 2025-09-16 17:18:55 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 17:18:55 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 17:18:55 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 17:18:55 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 17:18:55 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 23:20:03 INF Adding event: Scenario_benchmark-stop 2025-09-16 23:45:47 INF Stopping Lodemon 2025-09-16 23:45:47 INF Stopping monitoring service...