==================================================================================================== ============================= Pod describe lodemon-5c9b467df-6b9mr ============================= ==================================================================================================== Name: lodemon-5c9b467df-6b9mr Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Wed, 17 Sep 2025 04:58:43 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5c9b467df skaffold.dev/run-id=ae2f9e48-3a99-4676-b720-4c98f1a0f924 Annotations: Status: Running IP: 10.106.40.161 IPs: IP: 10.106.40.161 Controlled By: ReplicaSet/lodemon-5c9b467df Containers: lodemon: Container ID: containerd://c9b38485b927db370955e81bee06319891a9a07211e019b612efe1d2424bcfd9 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Wed, 17 Sep 2025 04:58:44 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fzcpq (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-fzcpq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== =============================== Pod logs lodemon-5c9b467df-6b9mr =============================== ==================================================================================================== 2025-09-17 04:58:44 INF Starting lodemon server on port 8080 2025-09-17 04:58:44 INF Initializing clients... 2025-09-17 04:58:44 DBG Fetching ID token for accessing prometheus 2025-09-17 04:58:44 DBG Fetched ID token successfully 2025-09-17 04:58:44 INF Clients initialized successfully 2025-09-17 04:59:20 INF Configuring Lodemon 2025-09-17 04:59:20 INF Configuring monitoring units 2025-09-17 04:59:20 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-17 04:59:20 TRC Parsing Prometheus queries from config 2025-09-17 04:59:20 TRC Parsing component query: DS Java heap mem usage 2025-09-17 04:59:20 TRC Parsing component query: IDM Java heap mem usage 2025-09-17 04:59:20 TRC Parsing component query: AM Java heap mem usage 2025-09-17 04:59:20 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-17 04:59:20 TRC Parsing global query: 2025-09-17 04:59:20 TRC Parsing scoped query: DS Identity backend connections 2025-09-17 04:59:20 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-17 04:59:20 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-17 04:59:20 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-17 04:59:20 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-17 04:59:20 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-17 04:59:20 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-17 04:59:20 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-17 04:59:20 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-17 04:59:20 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-17 04:59:20 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-17 04:59:20 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-17 04:59:20 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-17 04:59:20 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-17 04:59:20 INF  - All monitoring units configured 2025-09-17 04:59:20 INF Configuring monitoring service 2025-09-17 04:59:20 INF  - Adding monitoring units to the service 2025-09-17 04:59:20 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-17 04:59:20 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-17 04:59:20 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-17 04:59:20 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-17 04:59:20 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-17 04:59:20 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-17 04:59:20 INF  - All monitoring units added to the service 2025-09-17 04:59:20 INF Starting Lodemon now 2025-09-17 04:59:20 INF Starting monitoring service... 2025-09-17 04:59:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 04:59:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 04:59:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 04:59:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 04:59:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 04:59:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 04:59:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 04:59:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 04:59:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 04:59:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 04:59:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 04:59:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 04:59:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:22 INF Adding event: Scenario_benchmark 2025-09-17 04:59:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 04:59:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 04:59:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 04:59:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 04:59:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 04:59:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 04:59:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 04:59:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 04:59:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 04:59:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 04:59:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 04:59:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 04:59:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 04:59:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 04:59:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 04:59:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 04:59:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 04:59:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 04:59:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 04:59:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 04:59:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 04:59:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 04:59:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 04:59:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 04:59:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 04:59:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 04:59:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 04:59:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 04:59:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 04:59:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 04:59:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 04:59:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 04:59:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 04:59:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 04:59:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 04:59:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 04:59:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 04:59:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:00:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:00:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:00:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:00:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:00:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:00:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:00:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:00:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:00:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:00:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:00:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:00:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:00:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:00:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:00:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:00:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:00:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:00:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:00:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:00:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:00:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:00:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:00:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:00:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:00:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:00:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:00:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:00:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:00:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:00:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:00:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:00:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:00:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:00:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:00:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:00:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:00:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:00:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:00:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:00:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:00:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:00:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:00:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:00:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:00:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:00:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:00:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:00:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:00:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:00:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:00:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:00:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:00:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:00:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:00:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:00:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:00:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:00:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:00:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:00:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:00:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:00:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:00:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:00:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:00:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:00:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:00:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:00:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:01:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:01:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:01:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:01:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:01:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:01:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:01:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:01:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:01:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:01:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:01:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:01:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:01:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:01:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:01:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:01:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:01:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:01:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:01:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:01:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:01:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:01:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:01:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:01:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:01:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:01:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:01:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:01:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:01:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:01:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:01:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:01:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:01:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:01:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:01:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:01:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:01:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:01:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:01:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:01:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:01:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:01:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:01:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:01:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:01:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:01:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:01:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:01:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:01:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:01:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:01:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:01:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:01:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:01:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:01:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:01:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:01:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:01:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:01:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:01:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:01:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:01:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:01:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:01:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:01:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:01:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:01:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:01:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:02:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:02:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:02:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:02:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:02:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:02:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:02:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:02:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:02:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:02:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:02:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:02:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:02:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:02:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:02:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:02:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:02:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:02:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:02:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:02:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:02:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:02:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:02:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:02:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:02:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:02:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:02:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:02:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:02:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:02:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:02:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:02:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:02:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:02:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:02:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:02:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:02:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:02:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:02:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:02:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:02:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:02:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:02:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:02:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:02:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:02:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:02:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:02:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:02:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:02:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:02:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:02:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:02:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:02:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:02:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:02:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:02:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:02:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:02:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:02:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:02:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:02:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:02:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:02:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:02:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:02:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:02:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:02:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:03:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:03:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:03:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:03:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:03:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:03:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:03:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:03:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:03:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:03:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:03:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:03:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:03:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:03:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:03:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:03:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:03:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:03:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:03:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:03:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:03:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:03:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:03:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:03:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:03:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:03:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:03:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:03:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:03:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:03:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:03:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:03:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:03:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:03:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:03:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:03:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:03:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:03:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:03:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:03:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:03:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:03:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:03:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:03:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:03:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:03:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:03:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:03:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:03:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:03:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:03:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:03:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:03:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:03:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:03:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:03:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:03:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:03:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:03:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:03:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:03:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:03:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:03:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:03:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:03:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:03:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:03:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:03:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:03:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:04:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:04:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:04:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:04:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:04:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:04:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:04:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:04:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:04:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:04:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:04:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:04:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:04:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:04:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:04:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:04:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:04:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:04:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:04:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:04:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:04:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:04:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:04:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:04:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:04:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:04:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:04:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:04:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:04:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:04:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:04:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:04:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:04:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:04:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:04:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:04:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:04:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:04:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:04:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:04:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:04:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:04:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:04:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:04:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:04:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:04:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:04:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:04:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:04:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:04:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:04:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:04:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:04:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:04:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:04:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:04:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:04:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:04:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:04:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:04:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:04:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:04:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:04:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:04:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:04:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:04:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:04:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:04:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:04:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:04:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:04:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:04:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:04:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:04:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:05:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:05:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:05:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:05:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:05:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:05:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:05:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:05:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:05:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:05:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:05:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:05:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:05:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:05:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:05:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:05:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:05:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:05:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:05:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:05:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:05:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:05:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:05:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:05:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:05:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:05:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:05:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:05:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:05:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:05:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:05:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:05:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:05:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:05:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:05:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:05:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:05:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:05:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:05:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:05:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:05:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:05:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:05:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:05:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:05:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:05:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:05:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:05:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:05:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:05:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:05:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:05:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:05:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:05:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:05:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:05:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:05:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:05:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:05:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:05:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:05:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:05:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:05:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:05:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:05:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:05:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:05:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:05:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:05:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:05:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:05:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:05:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:05:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:05:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:06:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:06:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:06:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:06:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:06:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:06:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:06:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:06:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:06:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:06:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:06:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:06:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:06:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:06:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:06:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:06:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:06:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:06:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:06:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:06:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:06:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:06:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:06:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:06:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:06:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:06:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:06:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:06:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:06:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:06:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:06:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:06:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:06:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:06:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:06:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:06:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:06:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:06:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:06:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:06:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:06:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:06:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:06:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:06:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:06:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:06:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:06:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:06:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:06:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:06:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:06:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:06:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:06:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:06:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:06:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:06:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:06:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:06:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:06:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:06:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:06:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:06:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:06:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:06:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:06:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:06:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:06:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:06:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:06:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:06:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:06:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:06:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:06:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:07:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:07:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:07:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:07:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:07:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:07:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:07:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:07:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:07:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:07:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:07:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:07:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:07:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:07:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:07:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:07:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:07:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:07:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:07:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:07:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:07:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:07:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:07:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:07:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:07:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:07:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:07:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:07:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:07:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:07:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:07:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:07:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:07:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:07:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:07:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:07:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:07:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:07:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:07:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:07:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:07:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:07:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:07:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:07:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:07:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:07:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:07:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:07:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:07:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:07:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:07:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:07:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:07:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:07:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:07:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:07:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:07:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:07:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:07:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:07:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:07:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:07:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:07:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:07:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:07:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:07:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:07:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:07:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:07:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:07:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:07:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:07:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:07:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:07:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:08:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:08:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:08:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:08:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:08:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:08:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:08:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:08:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:08:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:08:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:08:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:08:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:08:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:08:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:08:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:08:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:08:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:08:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:08:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:08:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:08:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:08:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:08:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:08:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:08:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:08:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:08:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:08:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:08:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:08:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:08:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:08:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:08:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:08:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:08:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:08:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:08:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:08:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:08:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:08:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:08:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:08:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:08:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:08:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:08:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:08:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:08:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:08:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:08:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:08:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:08:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:08:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:08:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:08:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:08:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:08:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:08:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:08:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:08:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:08:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:08:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:08:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:08:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:08:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:08:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:08:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:08:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:08:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:08:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:09:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:09:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:09:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:09:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:09:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:09:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:09:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:09:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:09:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:09:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:09:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:09:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:09:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:09:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:09:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:09:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:09:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:09:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:09:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:09:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:09:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:09:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:09:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:09:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:09:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:09:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:09:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:09:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:09:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:09:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:09:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:09:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:09:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:09:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:09:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:09:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:09:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:09:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:09:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:09:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:09:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:09:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:09:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:09:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:09:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:09:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:09:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:09:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:09:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:09:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:09:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:09:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:09:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:09:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:09:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:09:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:09:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:09:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:09:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:09:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:09:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:09:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:09:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:09:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:09:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:09:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:09:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:09:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:09:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:09:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:09:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:10:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:10:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:10:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:10:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:10:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:10:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:10:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:10:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:10:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:10:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:10:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:10:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:10:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:10:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:10:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:10:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:10:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:10:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:10:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:10:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:10:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:10:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:10:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:10:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:10:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:10:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:10:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:10:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:10:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:10:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:10:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:10:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:10:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:10:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:10:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:10:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:10:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:10:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:10:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:10:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:10:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:10:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:10:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:10:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:10:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:10:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:10:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:10:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:10:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:10:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:10:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:10:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:10:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:10:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:10:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:10:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:10:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:10:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:10:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:10:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:10:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:10:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:10:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:10:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:10:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:11:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:11:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:11:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:11:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:11:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:11:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:11:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:11:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:11:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:11:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:11:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:11:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:11:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:11:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:11:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:11:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:11:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:11:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:11:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:11:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:11:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:11:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:11:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:11:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:11:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:11:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:11:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:11:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:11:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:11:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:11:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:11:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:11:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:11:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:11:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:11:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:11:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:11:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:11:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:11:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:11:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:11:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:11:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:11:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:11:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:11:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:11:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:11:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:11:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:11:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:11:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:11:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:11:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:11:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:11:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:11:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:11:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:11:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:11:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:11:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:11:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:11:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:11:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:11:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:11:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:11:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:11:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:11:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:11:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:11:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:11:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:11:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:11:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:11:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:12:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:12:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:12:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:12:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:12:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:12:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:12:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:12:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:12:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:12:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:12:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:12:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:12:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:12:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:12:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:12:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:12:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:12:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:12:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:12:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:12:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:12:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:12:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:12:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:12:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:12:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:12:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:12:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:12:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:12:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:12:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:12:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:12:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:12:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:12:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:12:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:12:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:12:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:12:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:12:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:12:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:12:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:12:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:12:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:12:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:12:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:12:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:12:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:12:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:12:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:12:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:12:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:12:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:12:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:12:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:12:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:12:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:12:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:12:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:12:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:12:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:12:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:12:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:12:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:12:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:12:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:12:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:12:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:12:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:12:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:12:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:12:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:12:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:13:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:13:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:13:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:13:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:13:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:13:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:13:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:13:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:13:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:13:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:13:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:13:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:13:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:13:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:13:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:13:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:13:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:13:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:13:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:13:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:13:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:13:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:13:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:13:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:13:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:13:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:13:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:13:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:13:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:13:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:13:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:13:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:13:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:13:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:13:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:13:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:13:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:13:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:13:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:13:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:13:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:13:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:13:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:13:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:13:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:13:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:13:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:13:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:13:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:13:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:13:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:13:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:13:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:13:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:13:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:13:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:13:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:13:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:13:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:13:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:13:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:13:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:13:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:13:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:13:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:13:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:13:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:13:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:14:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:14:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:14:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:14:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:14:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:14:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:14:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:14:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:14:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:14:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:14:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:14:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:14:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:14:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:14:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:14:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:14:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:14:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:14:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:14:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:14:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:14:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:14:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:14:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:14:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:14:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:14:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:14:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:14:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:14:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:14:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:14:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:14:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:14:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:14:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:14:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:14:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:14:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:14:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:14:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:14:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:14:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:14:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:14:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:14:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:14:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:14:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:14:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:14:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:14:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:14:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:14:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:14:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:14:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:14:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:14:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:14:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:14:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:14:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:14:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:14:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:14:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:14:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:14:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:14:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:14:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:14:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:14:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:14:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:14:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:14:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:14:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:14:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:14:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:15:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:15:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:15:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:15:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:15:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:15:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:15:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:15:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:15:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:15:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:15:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:15:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:15:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:15:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:15:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:15:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:15:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:15:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:15:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:15:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:15:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:15:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:15:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:15:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:15:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:15:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:15:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:15:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:15:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:15:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:15:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:15:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:15:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:15:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:15:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:15:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:15:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:15:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:15:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:15:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:15:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:15:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:15:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:15:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:15:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:15:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:15:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:15:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:15:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:15:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:15:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:15:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:15:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:15:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:15:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:15:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:15:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:15:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:15:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:15:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:15:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:15:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:15:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:15:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:15:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:15:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:15:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:15:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:15:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:15:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:15:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:15:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:15:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:15:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:16:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:16:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:16:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:16:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:16:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:16:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:16:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:16:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:16:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:16:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:16:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:16:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:16:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:16:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:16:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:16:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:16:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:16:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:16:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:16:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:16:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:16:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:16:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:16:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:16:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:16:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:16:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:16:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:16:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:16:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:16:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:16:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:16:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:16:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:16:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:16:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:16:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:16:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:16:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:16:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:16:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:16:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:16:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:16:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:16:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:16:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:16:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:16:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:16:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:16:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:16:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:16:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:16:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:16:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:16:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:16:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:16:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:16:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:16:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:16:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:16:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:16:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:16:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:16:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:16:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:16:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:16:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:16:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:17:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:17:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:17:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:17:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:17:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:17:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:17:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:17:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:17:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:17:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:17:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:17:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:17:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:17:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:17:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:17:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:17:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:17:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:17:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:17:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:17:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:17:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:17:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:17:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:17:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:17:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:17:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:17:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:17:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:17:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:17:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:17:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:17:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:17:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:17:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:17:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:17:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:17:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:17:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:17:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:17:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:17:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:17:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:17:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:17:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:17:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:17:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:17:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:17:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:17:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:17:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:17:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:17:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:17:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:17:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:17:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:17:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:17:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:17:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:17:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:17:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:17:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:17:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:17:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:17:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:17:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:17:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:17:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:18:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:18:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:18:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:18:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:18:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:18:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:18:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:18:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:18:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:18:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:18:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:18:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:18:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:18:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:18:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:18:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:18:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:18:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:18:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:18:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:18:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:18:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:18:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:18:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:18:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:18:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:18:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:18:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:18:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:18:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:18:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:18:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:18:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:18:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:18:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:18:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:18:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:18:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:18:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:18:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:18:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:18:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:18:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:18:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:18:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:18:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:18:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:18:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:18:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:18:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:18:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:18:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:18:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:18:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:18:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:18:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:18:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:18:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:18:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:18:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:18:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:18:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:18:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:18:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:18:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:18:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:18:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:18:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:18:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:19:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:19:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:19:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:19:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:19:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:19:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:19:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:19:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:19:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:19:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:19:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:19:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:19:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:19:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:19:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:19:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:19:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:19:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:19:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:19:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:19:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:19:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:19:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:19:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:19:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:19:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:19:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:19:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:19:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:19:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:19:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:19:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:19:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:19:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:19:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:19:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:19:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:19:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:19:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:19:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:19:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:19:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:19:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:19:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:19:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:19:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:19:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:19:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:19:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:19:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:19:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:19:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:19:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:19:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:19:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:19:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:19:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:19:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:19:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:19:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:19:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:19:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:19:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:19:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:19:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:19:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:19:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:19:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:19:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:19:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:19:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:19:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:19:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:19:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:20:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:20:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:20:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:20:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:20:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:20:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:20:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:20:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:20:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:20:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:20:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:20:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:20:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:20:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:20:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:20:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:20:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:20:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:20:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:20:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:20:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:20:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:20:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:20:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:20:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:20:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:20:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:20:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:20:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:20:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:20:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:20:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:20:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:20:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:20:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:20:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:20:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:20:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:20:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:20:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:20:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:20:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:20:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:20:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:20:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:20:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:20:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:20:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:20:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:20:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:20:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:20:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:20:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:20:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:20:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:20:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:20:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:20:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:20:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:20:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:20:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:20:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:20:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:20:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:20:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:20:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:20:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:20:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:20:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:20:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:20:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:20:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:20:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:21:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:21:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:21:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:21:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:21:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:21:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:21:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:21:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:21:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:21:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:21:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:21:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:21:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:21:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:21:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:21:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:21:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:21:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:21:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:21:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:21:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:21:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:21:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:21:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:21:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:21:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:21:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:21:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:21:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:21:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:21:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:21:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:21:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:21:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:21:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:21:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:21:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:21:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:21:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:21:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:21:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:21:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:21:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:21:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:21:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:21:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:21:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:21:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:21:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:21:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:21:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:21:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:21:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:21:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:21:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:21:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:21:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:21:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:21:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:21:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:21:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:21:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:22:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:22:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:22:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:22:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:22:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:22:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:22:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:22:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:22:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:22:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:22:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:22:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:22:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:22:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:22:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:22:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:22:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:22:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:22:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:22:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:22:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:22:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:22:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:22:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:22:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:22:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:22:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:22:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:22:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:22:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:22:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:22:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:22:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:22:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:22:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:22:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:22:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:22:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:22:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:22:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:22:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:22:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:22:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:22:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:22:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:22:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:22:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:22:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:22:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:22:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:22:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:22:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:22:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:22:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:22:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:22:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:22:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:22:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:22:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:22:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:22:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:22:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:22:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:22:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:22:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:22:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:22:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:22:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:23:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:23:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:23:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:23:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:23:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:23:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:23:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:23:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:23:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:23:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:23:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:23:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:23:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:23:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:23:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:23:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:23:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:23:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:23:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:23:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:23:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:23:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:23:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:23:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:23:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:23:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:23:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:23:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:23:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:23:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:23:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:23:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:23:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:23:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:23:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:23:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:23:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:23:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:23:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:23:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:23:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:23:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:23:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:23:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:23:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:23:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:23:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:23:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:23:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:23:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:23:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:23:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:23:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:23:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:23:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:23:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:23:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:23:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:23:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:23:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:23:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:23:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:23:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:23:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:23:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:23:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:23:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:23:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:24:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:24:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:24:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:24:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:24:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:24:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:24:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:24:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:24:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:24:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:24:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:24:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:24:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:24:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:24:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:24:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:24:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:24:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:24:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:24:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:24:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:24:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:24:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:24:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:24:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:24:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:24:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:24:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:24:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:24:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:24:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:24:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:24:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:24:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:24:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:24:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:24:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:24:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:24:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:24:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:24:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:24:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:24:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:24:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:24:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:24:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:24:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:24:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:24:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:24:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:24:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:24:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:24:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:24:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:24:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:24:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:24:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:24:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:24:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:24:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:24:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:24:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:24:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:24:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:24:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:24:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:24:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:24:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:25:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:25:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:25:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:25:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:25:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:25:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:25:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:25:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:25:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:25:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:25:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:25:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:25:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:25:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:25:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:25:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:25:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:25:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:25:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:25:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:25:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:25:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:25:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:25:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:25:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:25:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:25:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:25:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:25:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:25:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:25:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:25:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:25:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:25:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:25:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:25:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:25:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:25:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:25:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:25:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:25:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:25:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:25:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:25:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:25:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:25:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:25:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:25:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:25:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:25:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:25:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:25:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:25:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:25:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:25:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:25:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:25:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:25:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:25:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:25:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:25:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:25:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:25:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:25:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:25:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:25:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:25:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:25:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:26:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:26:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:26:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:26:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:26:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:26:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:26:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:26:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:26:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:26:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:26:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:26:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:26:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:26:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:26:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:26:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:26:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:26:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:26:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:26:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:26:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:26:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:26:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:26:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:26:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:26:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:26:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:26:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:26:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:26:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:26:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:26:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:26:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:26:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:26:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:26:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:26:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:26:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:26:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:26:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:26:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:26:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:26:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:26:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:26:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:26:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:26:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:26:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:26:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:26:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:26:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:26:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:26:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:26:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:26:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:26:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:26:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:26:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:26:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:26:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:26:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:26:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:26:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:26:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:26:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:26:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:26:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:26:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:26:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:27:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:27:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:27:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:27:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:27:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:27:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:27:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:27:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:27:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:27:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:27:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:27:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:27:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:27:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:27:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:27:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:27:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:27:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:27:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:27:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:27:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:27:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:27:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:27:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:27:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:27:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:27:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:27:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:27:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:27:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:27:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:27:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:27:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:27:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:27:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:27:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:27:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:27:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:27:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:27:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:27:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:27:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:27:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:27:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:27:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:27:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:27:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:27:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:27:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:27:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:27:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:27:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:27:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:27:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:27:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:27:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:27:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:27:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:27:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:27:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:27:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:27:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:27:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:27:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:27:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:27:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:27:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:27:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:27:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:27:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:27:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:27:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:27:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:27:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:28:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:28:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:28:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:28:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:28:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:28:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:28:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:28:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:28:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:28:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:28:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:28:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:28:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:28:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:28:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:28:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:28:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:28:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:28:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:28:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:28:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:28:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:28:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:28:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:28:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:28:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:28:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:28:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:28:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:28:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:28:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:28:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:28:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:28:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:28:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:28:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:28:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:28:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:28:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:28:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:28:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:28:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:28:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:28:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:28:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:28:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:28:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:28:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:28:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:28:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:28:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:28:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:28:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:28:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:28:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:28:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:28:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:28:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:28:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:28:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:28:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:28:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:28:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:28:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:28:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:28:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:28:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:28:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:28:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:28:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:28:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:28:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:28:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:28:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:29:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:29:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:29:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:29:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:29:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:29:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:29:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:29:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:29:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:29:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:29:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:29:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:29:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:29:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:29:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:29:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:29:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:29:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:29:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:29:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:29:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:29:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:29:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:29:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:29:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:29:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:29:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:29:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:29:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:29:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:29:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:29:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:29:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:29:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:29:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:29:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:29:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:29:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:29:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:29:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:29:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:29:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:29:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:29:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:29:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:29:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:29:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:29:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:29:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:29:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:29:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:29:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:29:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:29:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:29:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:29:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:29:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:29:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:29:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:29:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:29:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:29:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:29:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:29:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:29:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:29:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:29:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:29:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:29:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:29:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:29:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:29:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:29:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:29:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:30:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:30:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:30:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:30:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:30:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:30:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:30:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:30:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:30:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:30:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:30:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:30:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:30:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:30:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:30:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:30:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:30:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:30:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:30:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:30:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:30:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:30:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:30:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:30:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:30:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:30:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:30:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:30:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:30:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:30:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:30:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:30:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:30:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:30:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:30:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:30:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:30:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:30:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:30:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:30:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:30:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:30:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:30:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:30:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:30:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:30:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:30:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:30:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:30:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:30:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:30:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:30:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:30:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:30:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:30:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:30:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:30:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:30:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:30:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:30:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:30:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:30:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:30:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:30:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:30:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:30:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:30:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:30:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:30:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:30:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:30:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:30:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:30:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:30:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:31:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:31:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:31:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:31:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:31:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:31:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:31:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:31:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:31:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:31:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:31:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:31:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:31:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:31:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:31:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:31:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:31:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:31:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:31:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:31:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:31:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:31:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:31:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:31:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:31:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:31:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:31:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:31:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:31:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:31:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:31:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:31:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:31:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:31:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:31:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:31:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:31:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:31:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:31:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:31:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:31:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:31:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:31:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:31:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:31:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:31:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:31:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:31:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:31:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:31:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:31:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:31:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:31:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:31:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:31:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:31:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:31:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:31:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:31:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:31:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:31:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:31:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:31:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:31:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:31:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:31:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:31:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:31:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:31:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:31:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:31:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:31:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:31:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:32:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:32:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:32:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:32:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:32:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:32:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:32:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:32:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:32:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:32:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:32:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:32:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:32:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:32:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:32:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:32:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:32:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:32:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:32:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:32:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:32:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:32:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:32:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:32:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:32:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:32:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:32:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:32:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:32:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:32:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:32:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:32:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:32:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:32:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:32:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:32:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:32:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:32:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:32:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:32:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:32:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:32:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:32:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:32:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:32:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:32:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:32:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:32:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:32:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:32:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:32:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:32:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:32:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:32:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:32:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:32:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:32:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:32:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:32:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:32:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:32:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:32:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:32:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:32:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:32:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:32:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:32:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:32:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:33:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:33:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:33:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:33:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:33:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:33:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:33:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:33:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:33:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:33:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:33:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:33:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:33:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:33:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:33:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:33:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:33:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:33:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:33:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:33:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:33:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:33:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:33:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:33:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:33:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:33:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:33:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:33:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:33:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:33:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:33:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:33:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:33:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:33:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:33:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:33:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:33:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:33:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:33:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:33:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:33:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:33:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:33:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:33:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:33:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:33:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:33:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:33:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:33:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:33:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:33:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:33:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:33:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:33:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:33:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:33:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:33:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:33:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:33:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:33:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:33:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:33:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:33:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:33:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:33:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:33:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:33:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:33:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:34:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:34:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:34:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:34:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:34:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:34:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:34:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:34:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:34:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:34:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:34:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:34:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:34:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:34:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:34:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:34:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:34:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:34:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:34:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:34:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:34:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:34:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:34:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:34:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:34:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:34:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:34:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:34:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:34:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:34:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:34:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:34:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:34:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:34:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:34:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:34:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:34:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:34:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:34:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:34:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:34:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:34:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:34:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:34:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:34:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:34:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:34:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:34:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:34:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:34:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:34:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:34:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:34:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:34:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:34:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:34:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:34:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:34:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:34:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:34:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:34:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:34:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:34:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:34:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:34:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:34:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:34:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:34:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:34:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:34:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:34:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:34:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:34:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:34:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:35:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:35:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:35:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:35:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:35:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:35:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:35:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:35:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:35:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:35:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:35:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:35:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:35:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:35:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:35:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:35:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:35:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:35:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:35:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:35:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:35:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:35:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:35:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:35:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:35:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:35:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:35:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:35:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:35:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:35:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:35:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:35:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:35:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:35:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:35:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:35:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:35:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:35:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:35:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:35:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:35:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:35:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:35:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:35:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:35:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:35:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:35:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:35:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:35:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:35:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:35:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:35:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:35:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:35:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:35:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:35:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:35:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:35:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:35:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:35:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:35:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:35:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:35:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:35:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:35:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:35:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:35:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:35:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:35:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:36:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:36:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:36:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:36:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:36:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:36:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:36:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:36:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:36:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:36:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:36:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:36:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:36:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:36:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:36:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:36:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:36:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:36:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:36:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:36:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:36:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:36:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:36:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:36:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:36:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:36:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:36:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:36:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:36:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:36:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:36:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:36:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:36:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:36:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:36:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:36:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:36:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:36:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:36:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:36:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:36:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:36:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:36:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:36:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:36:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:36:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:36:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:36:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:36:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:36:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:36:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:36:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:36:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:36:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:36:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:36:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:36:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:36:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:36:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:36:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:36:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:36:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:36:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:36:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:36:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:36:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:36:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:36:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:36:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:36:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:36:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:36:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:36:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:36:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:37:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:37:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:37:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:37:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:37:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:37:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:37:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:37:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:37:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:37:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:37:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:37:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:37:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:37:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:37:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:37:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:37:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:37:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:37:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:37:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:37:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:37:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:37:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:37:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:37:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:37:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:37:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:37:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:37:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:37:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:37:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:37:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:37:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:37:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:37:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:37:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:37:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:37:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:37:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:37:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:37:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:37:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:37:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:37:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:37:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:37:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:37:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:37:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:37:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:37:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:37:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:37:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:37:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:37:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:37:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:37:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:37:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:37:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:37:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:37:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:37:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:37:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:37:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:37:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:37:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:37:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:37:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:37:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:37:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:37:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:37:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:37:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:37:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:37:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:38:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:38:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:38:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:38:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:38:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:38:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:38:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:38:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:38:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:38:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:38:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:38:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:38:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:38:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:38:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:38:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:38:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:38:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:38:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:38:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:38:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:38:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:38:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:38:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:38:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:38:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:38:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:38:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:38:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:38:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:38:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:38:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:38:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:38:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:38:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:38:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:38:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:38:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:38:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:38:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:38:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:38:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:38:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:38:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:38:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:38:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:38:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:38:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:38:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:38:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:38:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:38:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:38:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:38:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:38:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:38:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:38:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:38:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:38:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:38:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:38:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:38:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:38:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:38:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:38:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:38:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:38:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:38:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:38:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:38:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:38:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:38:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:38:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:38:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:39:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:39:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:39:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:39:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:39:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:39:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:39:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:39:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:39:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:39:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:39:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:39:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:39:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:39:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:39:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:39:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:39:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:39:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:39:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:39:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:39:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:39:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:39:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:39:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:39:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:39:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:39:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:39:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:39:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:39:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:39:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:39:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:39:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:39:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:39:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:39:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:39:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:39:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:39:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:39:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:39:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:39:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:39:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:39:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:39:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:39:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:39:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:39:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:39:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:39:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:39:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:39:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:39:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:39:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:39:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:39:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:39:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:39:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:39:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:39:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:39:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:39:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:40:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:40:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:40:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:40:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:40:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:40:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:40:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:40:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:40:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:40:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:40:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:40:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:40:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:40:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:40:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:40:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:40:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:40:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:40:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:40:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:40:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:40:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:40:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:40:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:40:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:40:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:40:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:40:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:40:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:40:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:40:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:40:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:40:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:40:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:40:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:40:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:40:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:40:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:40:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:40:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:40:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:40:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:40:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:40:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:40:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:40:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:40:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:40:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:40:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:40:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:40:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:40:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:40:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:40:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:40:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:40:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:40:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:40:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:40:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:40:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:40:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:40:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:40:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:40:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:40:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:40:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:40:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:40:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:40:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:40:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:40:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:40:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:40:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:40:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:41:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:41:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:41:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:41:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:41:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:41:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:41:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:41:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:41:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:41:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:41:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:41:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:41:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:41:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:41:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:41:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:41:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:41:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:41:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:41:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:41:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:41:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:41:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:41:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:41:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:41:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:41:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:41:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:41:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:41:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:41:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:41:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:41:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:41:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:41:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:41:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:41:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:41:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:41:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:41:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:41:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:41:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:41:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:41:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:41:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:41:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:41:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:41:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:41:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:41:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:41:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:41:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:41:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:41:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:41:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:41:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:41:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:41:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:41:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:41:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:41:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:41:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:41:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:41:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:41:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:41:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:41:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:41:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:41:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:41:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:41:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:41:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:41:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:42:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:42:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:42:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:42:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:42:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:42:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:42:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:42:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:42:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:42:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:42:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:42:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:42:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:42:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:42:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:42:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:42:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:42:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:42:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:42:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:42:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:42:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:42:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:42:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:42:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:42:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:42:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:42:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:42:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:42:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:42:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:42:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:42:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:42:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:42:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:42:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:42:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:42:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:42:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:42:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:42:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:42:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:42:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:42:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:42:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:42:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:42:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:42:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:42:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:42:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:42:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:42:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:42:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:42:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:42:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:42:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:42:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:42:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:42:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:42:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:42:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:42:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:42:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:42:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:42:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:42:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:42:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:42:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:43:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:43:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:43:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:43:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:43:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:43:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:43:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:43:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:43:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:43:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:43:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:43:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:43:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:43:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:43:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:43:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:43:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:43:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:43:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:43:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:43:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:43:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:43:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:43:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:43:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:43:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:43:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:43:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:43:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:43:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:43:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:43:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:43:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:43:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:43:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:43:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:43:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:43:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:43:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:43:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:43:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:43:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:43:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:43:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:43:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:43:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:43:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:43:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:43:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:43:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:43:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:43:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:43:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:43:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:43:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:43:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:43:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:43:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:43:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:43:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:43:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:43:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:43:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:43:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:43:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:43:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:43:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:43:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:43:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:44:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:44:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:44:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:44:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:44:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:44:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:44:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:44:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:44:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:44:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:44:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:44:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:44:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:44:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:44:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:44:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:44:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:44:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:44:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:44:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:44:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:44:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:44:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:44:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:44:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:44:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:44:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:44:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:44:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:44:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:44:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:44:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:44:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:44:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:44:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:44:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:44:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:44:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:44:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:44:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:44:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:44:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:44:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:44:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:44:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:44:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:44:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:44:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:44:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:44:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:44:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:44:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:44:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:44:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:44:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:44:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:44:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:44:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:44:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:44:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:44:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:44:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:44:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:44:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:44:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:44:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:44:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:44:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:44:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:44:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:44:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:44:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:44:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:44:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:45:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:45:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:45:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:45:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:45:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:45:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:45:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:45:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:45:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:45:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:45:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:45:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:45:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:45:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:45:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:45:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:45:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:45:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:45:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:45:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:45:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:45:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:45:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:45:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:45:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:45:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:45:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:45:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:45:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:45:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:45:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:45:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:45:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:45:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:45:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:45:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:45:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:45:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:45:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:45:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:45:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:45:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:45:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:45:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:45:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:45:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:45:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:45:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:45:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:45:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:45:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:45:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:45:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:45:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:45:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:45:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:45:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:45:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:45:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:45:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:45:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:45:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:45:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:45:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:45:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:45:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:45:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:45:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:45:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:45:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:45:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:45:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:45:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:45:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:46:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:46:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:46:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:46:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:46:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:46:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:46:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:46:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:46:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:46:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:46:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:46:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:46:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:46:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:46:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:46:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:46:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:46:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:46:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:46:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:46:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:46:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:46:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:46:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:46:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:46:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:46:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:46:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:46:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:46:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:46:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:46:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:46:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:46:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:46:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:46:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:46:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:46:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:46:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:46:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:46:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:46:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:46:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:46:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:46:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:46:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:46:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:46:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:46:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:46:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:46:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:46:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:46:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:46:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:46:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:46:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:46:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:46:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:46:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:46:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:46:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:46:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:46:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:46:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:46:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:46:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:46:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:46:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:46:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:46:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:46:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:46:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:46:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:46:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:47:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:47:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:47:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:47:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:47:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:47:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:47:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:47:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:47:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:47:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:47:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:47:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:47:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:47:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:47:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:47:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:47:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:47:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:47:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:47:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:47:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:47:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:47:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:47:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:47:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:47:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:47:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:47:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:47:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:47:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:47:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:47:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:47:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:47:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:47:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:47:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:47:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:47:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:47:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:47:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:47:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:47:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:47:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:47:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:47:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:47:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:47:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:47:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:47:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:47:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:47:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:47:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:47:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:47:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:47:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:47:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:47:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:47:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:47:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:47:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:47:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:47:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:47:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:47:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:47:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:47:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:47:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:47:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:47:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:47:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:47:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:47:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:47:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:48:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:48:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:48:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:48:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:48:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:48:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:48:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:48:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:48:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:48:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:48:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:48:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:48:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:48:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:48:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:48:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:48:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:48:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:48:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:48:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:48:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:48:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:48:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:48:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:48:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:48:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:48:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:48:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:48:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:48:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:48:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:48:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:48:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:48:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:48:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:48:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:48:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:48:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:48:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:48:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:48:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:48:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:48:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:48:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:48:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:48:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:48:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:48:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:48:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:48:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:48:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:48:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:48:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:48:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:48:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:48:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:48:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:48:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:48:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:48:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:48:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:48:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:48:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:48:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:48:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:48:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:48:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:48:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:48:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:48:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:48:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:48:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:48:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:48:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:49:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:49:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:49:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:49:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:49:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:49:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:49:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:49:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:49:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:49:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:49:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:49:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:49:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:49:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:49:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:49:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:49:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:49:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:49:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:49:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:49:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:49:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:49:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:49:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:49:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:49:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:49:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:49:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:49:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:49:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:49:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:49:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:49:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:49:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:49:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:49:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:49:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:49:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:49:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:49:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:49:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:49:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:49:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:49:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:49:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:49:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:49:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:49:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:49:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:49:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:49:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:49:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:49:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:49:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:49:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:49:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:49:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:49:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:49:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:49:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:49:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:49:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:49:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:49:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:49:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:49:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:49:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:49:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:49:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:49:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:49:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:49:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:49:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:49:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:50:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:50:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:50:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:50:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:50:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:50:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:50:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:50:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:50:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:50:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:50:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:50:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:50:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:50:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:50:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:50:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:50:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:50:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:50:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:50:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:50:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:50:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:50:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:50:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:50:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:50:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:50:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:50:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:50:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:50:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:50:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:50:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:50:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:50:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:50:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:50:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:50:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:50:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:50:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:50:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:50:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:50:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:50:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:50:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:50:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:50:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:50:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:50:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:50:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:50:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:50:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:50:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:50:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:50:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:50:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:50:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:50:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:50:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:50:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:50:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:50:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:50:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:50:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:50:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:50:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:50:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:50:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:50:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:50:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:50:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:50:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:50:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:50:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:50:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:51:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:51:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:51:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:51:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:51:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:51:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:51:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:51:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:51:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:51:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:51:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:51:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:51:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:51:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:51:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:51:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:51:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:51:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:51:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:51:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:51:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:51:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:51:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:51:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:51:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:51:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:51:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:51:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:51:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:51:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:51:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:51:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:51:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:51:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:51:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:51:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:51:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:51:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:51:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:51:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:51:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:51:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:51:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:51:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:51:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:51:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:51:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:51:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:51:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:51:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:51:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:51:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:51:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:51:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:51:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:51:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:51:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:51:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:51:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:51:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:51:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:51:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:51:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:51:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:51:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:51:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:51:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:51:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:52:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:52:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:52:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:52:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:52:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:52:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:52:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:52:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:52:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:52:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:52:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:52:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:52:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:52:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:52:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:52:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:52:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:52:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:52:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:52:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:52:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:52:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:52:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:52:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:52:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:52:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:52:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:52:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:52:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:52:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:52:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:52:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:52:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:52:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:52:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:52:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:52:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:52:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:52:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:52:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:52:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:52:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:52:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:52:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:52:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:52:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:52:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:52:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:52:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:52:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:52:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:52:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:52:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:52:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:52:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:52:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:52:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:52:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:52:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:52:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:52:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:52:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:52:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:52:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:52:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:52:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:52:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:52:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:53:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:53:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:53:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:53:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:53:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:53:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:53:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:53:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:53:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:53:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:53:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:53:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:53:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:53:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:53:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:53:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:53:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:53:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:53:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:53:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:53:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:53:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:53:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:53:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:53:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:53:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:53:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:53:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:53:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:53:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:53:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:53:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:53:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:53:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:53:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:53:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:53:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:53:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:53:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:53:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:53:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:53:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:53:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:53:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:53:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:53:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:53:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:53:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:53:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:53:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:53:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:53:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:53:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:53:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:53:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:53:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:53:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:53:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:53:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:53:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:53:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:53:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:54:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:54:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:54:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:54:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:54:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:54:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:54:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:54:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:54:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:54:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:54:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:54:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:54:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:54:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:54:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:54:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:54:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:54:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:54:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:54:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:54:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:54:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:54:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:54:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:54:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:54:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:54:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:54:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:54:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:54:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:54:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:54:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:54:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:54:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:54:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:54:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:54:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:54:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:54:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:54:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:54:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:54:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:54:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:54:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:54:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:54:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:54:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:54:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:54:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:54:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:54:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:54:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:54:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:54:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:54:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:54:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:54:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:54:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:54:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:54:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:54:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:54:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:54:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:54:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:54:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:54:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:54:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:54:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:54:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:54:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:54:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:54:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:54:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:54:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:55:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:55:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:55:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:55:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:55:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:55:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:55:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:55:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:55:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:55:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:55:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:55:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:55:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:55:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:55:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:55:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:55:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:55:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:55:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:55:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:55:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:55:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:55:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:55:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:55:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:55:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:55:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:55:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:55:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:55:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:55:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:55:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:55:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:55:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:55:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:55:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:55:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:55:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:55:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:55:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:55:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:55:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:55:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:55:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:55:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:55:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:55:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:55:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:55:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:55:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:55:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:55:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:55:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:55:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:55:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:55:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:55:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:55:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:55:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:55:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:55:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:55:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:55:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:55:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:55:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:55:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:55:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:55:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:55:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:56:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:56:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:56:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:56:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:56:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:56:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:56:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:56:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:56:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:56:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:56:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:56:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:56:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:56:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:56:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:56:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:56:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:56:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:56:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:56:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:56:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:56:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:56:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:56:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:56:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:56:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:56:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:56:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:56:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:56:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:56:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:56:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:56:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:56:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:56:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:56:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:56:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:56:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:56:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:56:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:56:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:56:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:56:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:56:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:56:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:56:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:56:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:56:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:56:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:56:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:56:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:56:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:56:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:56:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:56:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:56:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:56:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:56:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:56:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:56:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:56:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:56:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:56:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:56:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:56:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:56:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:56:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:56:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:56:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:56:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:56:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:56:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:56:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:57:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:57:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:57:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:57:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:57:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:57:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:57:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:57:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:57:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:57:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:57:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:57:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:57:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:57:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:57:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:57:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:57:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:57:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:57:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:57:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:57:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:57:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:57:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:57:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:57:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:57:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:57:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:57:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:57:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:57:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:57:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:57:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:57:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:57:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:57:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:57:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:57:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:57:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:57:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:57:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:57:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:57:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:57:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:57:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:57:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:57:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:57:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:57:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:57:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:57:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:57:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:57:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:57:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:57:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:57:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:57:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:57:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:57:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:57:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:57:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:57:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:57:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:57:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:57:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:57:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:57:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:57:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:57:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:57:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:58:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:58:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:58:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:58:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:58:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:58:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:58:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:58:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:58:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:58:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:58:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:58:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:58:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:58:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:58:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:58:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:58:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:58:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:58:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:58:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:58:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:58:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:58:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:58:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:58:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:58:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:58:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:58:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:58:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:58:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:58:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:58:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:58:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:58:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:58:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:58:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:58:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:58:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:58:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:58:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:58:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:58:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:58:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:58:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:58:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:58:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:58:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:58:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:58:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:58:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:58:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:58:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:58:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:58:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:58:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:58:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:58:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:58:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:58:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:58:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:58:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:58:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:58:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:58:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:58:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:58:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:58:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:58:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:58:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:58:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:58:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:58:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:58:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:58:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:59:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:59:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:59:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:59:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:59:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:59:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:59:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:59:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:59:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:59:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:59:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:59:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:59:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:59:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:59:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:59:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:59:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:59:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:59:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:59:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:59:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:59:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:59:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:59:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:59:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:59:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:59:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:59:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:59:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:59:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:59:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:59:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:59:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:59:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:59:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:59:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:59:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:59:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:59:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:59:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:59:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:59:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:59:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:59:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:59:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:59:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:59:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:59:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:59:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:59:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:59:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:59:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:59:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:59:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:59:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:59:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:59:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 05:59:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:59:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:59:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 05:59:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 05:59:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 05:59:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 05:59:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 05:59:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 05:59:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 05:59:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 05:59:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 05:59:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 05:59:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 05:59:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 05:59:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 05:59:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:00:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:00:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:00:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:00:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:00:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:00:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:00:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:00:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:00:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:00:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:00:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:00:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:00:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:00:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:00:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:00:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:00:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:00:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:00:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:00:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:00:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:00:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:00:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:00:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:00:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:00:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:00:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:00:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:00:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:00:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:00:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:00:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:00:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:00:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:00:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:00:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:00:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:00:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:00:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:00:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:00:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:00:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:00:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:00:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:00:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:00:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:00:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:00:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:00:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:00:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:00:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:00:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:00:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:00:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:00:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:00:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:00:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:00:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:00:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:00:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:00:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:00:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:00:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:00:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:00:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:00:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:00:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:00:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:01:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:01:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:01:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:01:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:01:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:01:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:01:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:01:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:01:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:01:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:01:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:01:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:01:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:01:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:01:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:01:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:01:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:01:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:01:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:01:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:01:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:01:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:01:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:01:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:01:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:01:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:01:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:01:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:01:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:01:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:01:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:01:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:01:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:01:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:01:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:01:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:01:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:01:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:01:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:01:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:01:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:01:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:01:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:01:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:01:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:01:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:01:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:01:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:01:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:01:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:01:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:01:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:01:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:01:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:01:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:01:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:01:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:01:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:01:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:01:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:01:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:01:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:01:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:01:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:01:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:01:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:01:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:01:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:01:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:02:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:02:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:02:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:02:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:02:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:02:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:02:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:02:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:02:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:02:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:02:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:02:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:02:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:02:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:02:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:02:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:02:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:02:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:02:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:02:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:02:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:02:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:02:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:02:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:02:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:02:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:02:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:02:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:02:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:02:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:02:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:02:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:02:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:02:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:02:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:02:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:02:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:02:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:02:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:02:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:02:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:02:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:02:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:02:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:02:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:02:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:02:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:02:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:02:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:02:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:02:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:02:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:02:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:02:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:02:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:02:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:02:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:02:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:02:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:02:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:02:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:02:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:02:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:02:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:02:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:02:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:02:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:02:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:02:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:02:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:02:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:02:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:02:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:03:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:03:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:03:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:03:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:03:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:03:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:03:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:03:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:03:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:03:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:03:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:03:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:03:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:03:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:03:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:03:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:03:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:03:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:03:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:03:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:03:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:03:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:03:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:03:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:03:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:03:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:03:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:03:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:03:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:03:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:03:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:03:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:03:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:03:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:03:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:03:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:03:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:03:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:03:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:03:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:03:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:03:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:03:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:03:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:03:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:03:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-17 06:03:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-17 06:03:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-17 06:03:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-17 06:03:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-17 06:03:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-17 06:03:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-17 06:03:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-17 06:03:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-17 06:03:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-17 06:03:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-17 06:03:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-17 06:03:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-17 06:03:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-17 06:03:45 WRN Prometheus returned 401. Re-authenticating... 2025-09-17 06:03:45 INF Re-authenticating Prometheus client 2025-09-17 06:03:45 WRN Prometheus returned 401. Re-authenticating... 2025-09-17 06:03:45 WRN Prometheus returned 401. Re-authenticating... 2025-09-17 06:03:45 WRN Prometheus returned 401. Re-authenticating... 2025-09-17 06:03:45 WRN Prometheus returned 401. Re-authenticating... 2025-09-17 06:03:45 WRN Prometheus returned 401. Re-authenticating... 2025-09-17 10:53:42 INF Adding event: Scenario_benchmark-stop 2025-09-17 11:17:31 INF Stopping Lodemon 2025-09-17 11:17:31 INF Stopping monitoring service...