==================================================================================================== ============================ Pod describe lodemon-84c975fb49-gwm4k ============================ ==================================================================================================== Name: lodemon-84c975fb49-gwm4k Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Mon, 22 Sep 2025 00:53:12 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=84c975fb49 skaffold.dev/run-id=ad64f3bf-8f74-4705-8e21-9082623cb12d Annotations: Status: Running IP: 10.106.40.167 IPs: IP: 10.106.40.167 Controlled By: ReplicaSet/lodemon-84c975fb49 Containers: lodemon: Container ID: containerd://e6b53d7897f5f25398e8bf93f22f4e0d3585422e6a4ed639dbf34a661c19343d Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Mon, 22 Sep 2025 00:53:13 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cxlhb (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-cxlhb: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-84c975fb49-gwm4k ============================== ==================================================================================================== 2025-09-22 00:53:13 INF Starting lodemon server on port 8080 2025-09-22 00:53:13 INF Initializing clients... 2025-09-22 00:53:13 DBG Fetching ID token for accessing prometheus 2025-09-22 00:53:13 DBG Fetched ID token successfully 2025-09-22 00:53:13 INF Clients initialized successfully 2025-09-22 00:53:47 INF Configuring Lodemon 2025-09-22 00:53:47 INF Configuring monitoring units 2025-09-22 00:53:47 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-22 00:53:47 TRC Parsing Prometheus queries from config 2025-09-22 00:53:47 TRC Parsing component query: DS Java heap mem usage 2025-09-22 00:53:47 TRC Parsing component query: IDM Java heap mem usage 2025-09-22 00:53:47 TRC Parsing component query: AM Java heap mem usage 2025-09-22 00:53:47 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-22 00:53:47 TRC Parsing global query: 2025-09-22 00:53:47 TRC Parsing scoped query: DS Identity backend connections 2025-09-22 00:53:47 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-22 00:53:47 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-22 00:53:47 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-22 00:53:47 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-22 00:53:47 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-22 00:53:47 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-22 00:53:47 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-22 00:53:47 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-22 00:53:47 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-22 00:53:47 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-22 00:53:47 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-22 00:53:47 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-22 00:53:47 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-22 00:53:47 INF  - All monitoring units configured 2025-09-22 00:53:47 INF Configuring monitoring service 2025-09-22 00:53:47 INF  - Adding monitoring units to the service 2025-09-22 00:53:47 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-22 00:53:47 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-22 00:53:47 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-22 00:53:47 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-22 00:53:47 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-22 00:53:47 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-22 00:53:47 INF  - All monitoring units added to the service 2025-09-22 00:53:47 INF Starting Lodemon now 2025-09-22 00:53:47 INF Starting monitoring service... 2025-09-22 00:53:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:53:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:53:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:53:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:53:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:53:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:53:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:53:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:53:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:53:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:53:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:53:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:53:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:53:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:53:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:53:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:53:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:53:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:53:56 INF Adding event: Scenario_benchmark 2025-09-22 00:53:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:53:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:53:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:53:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:53:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:53:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:53:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:53:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:53:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:53:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:53:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:53:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:53:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:53:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:53:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:53:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:53:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:53:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:54:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:54:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:54:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:54:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:54:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:54:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:54:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:54:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:54:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:54:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:54:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:54:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:54:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:54:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:54:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:54:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:54:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:54:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:54:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:54:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:54:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:54:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:54:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:54:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:54:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:54:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:54:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:54:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:54:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:54:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:54:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:54:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:54:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:54:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:54:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:54:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:54:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:54:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:54:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:54:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:54:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:54:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:54:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:54:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:54:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:54:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:54:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:54:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:54:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:54:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:54:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:54:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:54:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:54:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:54:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:54:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:54:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:54:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:54:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:54:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:54:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:54:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:54:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:54:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:54:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:54:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:54:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:54:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:54:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:54:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:54:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:54:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:54:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:54:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:55:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:55:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:55:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:55:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:55:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:55:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:55:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:55:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:55:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:55:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:55:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:55:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:55:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:55:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:55:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:55:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:55:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:55:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:55:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:55:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:55:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:55:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:55:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:55:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:55:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:55:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:55:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:55:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:55:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:55:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:55:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:55:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:55:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:55:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:55:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:55:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:55:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:55:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:55:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:55:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:55:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:55:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:55:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:55:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:55:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:55:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:55:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:55:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:55:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:55:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:55:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:55:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:55:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:55:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:55:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:55:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:55:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:55:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:55:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:55:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:55:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:55:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:55:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:55:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:55:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:55:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:55:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:55:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:55:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:55:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:55:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:55:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:55:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:56:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:56:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:56:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:56:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:56:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:56:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:56:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:56:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:56:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:56:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:56:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:56:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:56:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:56:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:56:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:56:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:56:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:56:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:56:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:56:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:56:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:56:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:56:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:56:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:56:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:56:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:56:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:56:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:56:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:56:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:56:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:56:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:56:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:56:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:56:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:56:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:56:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:56:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:56:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:56:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:56:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:56:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:56:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:56:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:56:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:56:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:56:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:56:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:56:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:56:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:56:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:56:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:56:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:56:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:56:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:56:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:56:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:56:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:56:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:56:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:56:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:56:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:57:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:57:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:57:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:57:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:57:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:57:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:57:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:57:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:57:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:57:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:57:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:57:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:57:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:57:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:57:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:57:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:57:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:57:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:57:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:57:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:57:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:57:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:57:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:57:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:57:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:57:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:57:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:57:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:57:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:57:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:57:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:57:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:57:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:57:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:57:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:57:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:57:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:57:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:57:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:57:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:57:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:57:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:57:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:57:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:57:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:57:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:57:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:57:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:57:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:57:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:57:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:57:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:57:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:57:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:57:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:57:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:57:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:57:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:57:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:57:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:57:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:57:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:57:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:57:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:57:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:57:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:57:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:57:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:58:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:58:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:58:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:58:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:58:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:58:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:58:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:58:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:58:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:58:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:58:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:58:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:58:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:58:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:58:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:58:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:58:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:58:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:58:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:58:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:58:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:58:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:58:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:58:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:58:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:58:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:58:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:58:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:58:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:58:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:58:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:58:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:58:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:58:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:58:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:58:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:58:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:58:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:58:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:58:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:58:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:58:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:58:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:58:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:58:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:58:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:58:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:58:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:58:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:58:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:58:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:58:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:58:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:58:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:58:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:58:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:58:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:58:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:58:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:58:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:58:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:58:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:58:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:58:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:58:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:58:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:58:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:58:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:59:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:59:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:59:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:59:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:59:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:59:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:59:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:59:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:59:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:59:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:59:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:59:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:59:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:59:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:59:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:59:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:59:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:59:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:59:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:59:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:59:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:59:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:59:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:59:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:59:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:59:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:59:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:59:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:59:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:59:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:59:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:59:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:59:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:59:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:59:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:59:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:59:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:59:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:59:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:59:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:59:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:59:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:59:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:59:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:59:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:59:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:59:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:59:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:59:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:59:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:59:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:59:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:59:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:59:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:59:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 00:59:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 00:59:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 00:59:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 00:59:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 00:59:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 00:59:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 00:59:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 00:59:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 00:59:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 00:59:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 00:59:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 00:59:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 00:59:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 00:59:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:00:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:00:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:00:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:00:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:00:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:00:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:00:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:00:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:00:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:00:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:00:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:00:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:00:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:00:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:00:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:00:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:00:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:00:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:00:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:00:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:00:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:00:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:00:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:00:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:00:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:00:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:00:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:00:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:00:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:00:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:00:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:00:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:00:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:00:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:00:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:00:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:00:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:00:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:00:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:00:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:00:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:00:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:00:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:00:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:00:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:00:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:00:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:00:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:00:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:00:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:00:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:00:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:00:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:00:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:00:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:00:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:00:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:00:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:00:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:00:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:00:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:00:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:00:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:00:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:00:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:00:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:00:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:00:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:00:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:00:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:00:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:00:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:00:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:00:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:01:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:01:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:01:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:01:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:01:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:01:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:01:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:01:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:01:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:01:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:01:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:01:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:01:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:01:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:01:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:01:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:01:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:01:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:01:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:01:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:01:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:01:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:01:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:01:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:01:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:01:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:01:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:01:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:01:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:01:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:01:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:01:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:01:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:01:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:01:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:01:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:01:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:01:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:01:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:01:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:01:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:01:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:01:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:01:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:01:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:01:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:01:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:01:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:01:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:01:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:01:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:01:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:01:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:01:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:01:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:01:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:01:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:01:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:01:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:01:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:01:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:01:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:01:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:01:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:01:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:01:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:01:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:01:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:01:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:01:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:01:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:01:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:01:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:02:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:02:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:02:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:02:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:02:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:02:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:02:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:02:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:02:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:02:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:02:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:02:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:02:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:02:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:02:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:02:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:02:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:02:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:02:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:02:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:02:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:02:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:02:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:02:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:02:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:02:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:02:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:02:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:02:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:02:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:02:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:02:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:02:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:02:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:02:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:02:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:02:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:02:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:02:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:02:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:02:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:02:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:02:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:02:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:02:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:02:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:02:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:02:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:02:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:02:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:02:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:02:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:02:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:02:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:02:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:02:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:02:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:02:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:02:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:02:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:02:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:02:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:02:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:02:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:02:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:02:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:02:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:02:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:02:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:03:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:03:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:03:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:03:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:03:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:03:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:03:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:03:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:03:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:03:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:03:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:03:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:03:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:03:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:03:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:03:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:03:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:03:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:03:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:03:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:03:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:03:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:03:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:03:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:03:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:03:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:03:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:03:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:03:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:03:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:03:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:03:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:03:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:03:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:03:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:03:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:03:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:03:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:03:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:03:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:03:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:03:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:03:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:03:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:03:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:03:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:03:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:03:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:03:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:03:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:03:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:03:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:03:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:03:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:03:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:03:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:03:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:03:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:03:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:03:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:03:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:03:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:03:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:03:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:03:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:03:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:03:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:03:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:03:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:03:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:03:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:03:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:03:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:04:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:04:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:04:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:04:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:04:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:04:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:04:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:04:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:04:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:04:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:04:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:04:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:04:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:04:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:04:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:04:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:04:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:04:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:04:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:04:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:04:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:04:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:04:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:04:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:04:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:04:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:04:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:04:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:04:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:04:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:04:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:04:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:04:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:04:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:04:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:04:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:04:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:04:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:04:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:04:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:04:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:04:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:04:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:04:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:04:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:04:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:04:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:04:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:04:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:04:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:04:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:04:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:04:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:04:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:04:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:04:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:04:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:04:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:04:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:04:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:04:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:04:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:04:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:04:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:04:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:04:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:04:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:04:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:04:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:05:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:05:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:05:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:05:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:05:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:05:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:05:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:05:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:05:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:05:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:05:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:05:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:05:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:05:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:05:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:05:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:05:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:05:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:05:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:05:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:05:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:05:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:05:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:05:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:05:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:05:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:05:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:05:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:05:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:05:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:05:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:05:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:05:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:05:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:05:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:05:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:05:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:05:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:05:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:05:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:05:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:05:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:05:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:05:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:05:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:05:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:05:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:05:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:05:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:05:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:05:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:05:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:05:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:05:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:05:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:05:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:05:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:05:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:05:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:05:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:05:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:05:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:05:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:05:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:05:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:05:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:05:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:05:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:05:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:05:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:05:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:05:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:05:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:06:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:06:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:06:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:06:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:06:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:06:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:06:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:06:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:06:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:06:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:06:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:06:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:06:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:06:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:06:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:06:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:06:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:06:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:06:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:06:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:06:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:06:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:06:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:06:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:06:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:06:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:06:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:06:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:06:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:06:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:06:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:06:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:06:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:06:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:06:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:06:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:06:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:06:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:06:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:06:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:06:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:06:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:06:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:06:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:06:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:06:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:06:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:06:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:06:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:06:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:06:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:06:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:06:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:06:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:06:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:06:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:06:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:06:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:06:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:06:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:06:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:06:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:06:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:06:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:06:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:06:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:06:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:06:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:07:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:07:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:07:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:07:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:07:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:07:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:07:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:07:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:07:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:07:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:07:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:07:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:07:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:07:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:07:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:07:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:07:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:07:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:07:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:07:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:07:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:07:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:07:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:07:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:07:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:07:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:07:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:07:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:07:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:07:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:07:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:07:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:07:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:07:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:07:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:07:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:07:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:07:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:07:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:07:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:07:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:07:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:07:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:07:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:07:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:07:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:07:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:07:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:07:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:07:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:07:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:07:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:07:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:07:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:07:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:07:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:07:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:07:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:07:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:07:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:07:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:07:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:07:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:07:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:07:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:07:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:07:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:07:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:07:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:08:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:08:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:08:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:08:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:08:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:08:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:08:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:08:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:08:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:08:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:08:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:08:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:08:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:08:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:08:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:08:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:08:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:08:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:08:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:08:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:08:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:08:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:08:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:08:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:08:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:08:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:08:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:08:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:08:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:08:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:08:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:08:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:08:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:08:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:08:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:08:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:08:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:08:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:08:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:08:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:08:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:08:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:08:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:08:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:08:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:08:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:08:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:08:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:08:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:08:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:08:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:08:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:08:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:08:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:08:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:08:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:08:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:08:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:08:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:08:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:08:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:08:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:08:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:08:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:08:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:08:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:08:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:08:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:08:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:08:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:08:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:08:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:08:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:09:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:09:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:09:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:09:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:09:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:09:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:09:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:09:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:09:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:09:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:09:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:09:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:09:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:09:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:09:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:09:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:09:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:09:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:09:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:09:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:09:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:09:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:09:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:09:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:09:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:09:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:09:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:09:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:09:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:09:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:09:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:09:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:09:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:09:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:09:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:09:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:09:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:09:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:09:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:09:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:09:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:09:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:09:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:09:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:09:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:09:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:09:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:09:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:09:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:09:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:09:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:09:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:09:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:09:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:09:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:09:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:09:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:09:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:09:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:09:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:09:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:09:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:09:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:09:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:09:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:09:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:09:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:09:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:10:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:10:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:10:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:10:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:10:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:10:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:10:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:10:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:10:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:10:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:10:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:10:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:10:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:10:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:10:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:10:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:10:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:10:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:10:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:10:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:10:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:10:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:10:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:10:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:10:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:10:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:10:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:10:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:10:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:10:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:10:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:10:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:10:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:10:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:10:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:10:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:10:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:10:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:10:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:10:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:10:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:10:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:10:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:10:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:10:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:10:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:10:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:10:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:10:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:10:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:10:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:10:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:10:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:10:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:10:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:10:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:10:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:10:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:10:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:10:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:10:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:10:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:10:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:10:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:10:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:10:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:10:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:10:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:10:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:10:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:10:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:11:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:11:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:11:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:11:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:11:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:11:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:11:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:11:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:11:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:11:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:11:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:11:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:11:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:11:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:11:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:11:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:11:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:11:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:11:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:11:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:11:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:11:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:11:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:11:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:11:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:11:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:11:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:11:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:11:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:11:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:11:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:11:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:11:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:11:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:11:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:11:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:11:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:11:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:11:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:11:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:11:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:11:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:11:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:11:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:11:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:11:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:11:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:11:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:11:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:11:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:11:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:11:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:11:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:11:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:11:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:11:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:11:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:11:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:11:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:11:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:11:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:11:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:11:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:11:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:11:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:12:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:12:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:12:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:12:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:12:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:12:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:12:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:12:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:12:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:12:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:12:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:12:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:12:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:12:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:12:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:12:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:12:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:12:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:12:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:12:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:12:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:12:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:12:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:12:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:12:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:12:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:12:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:12:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:12:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:12:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:12:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:12:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:12:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:12:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:12:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:12:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:12:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:12:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:12:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:12:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:12:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:12:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:12:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:12:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:12:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:12:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:12:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:12:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:12:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:12:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:12:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:12:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:12:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:12:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:12:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:12:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:12:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:12:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:12:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:12:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:12:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:12:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:12:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:12:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:12:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:12:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:12:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:12:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:12:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:13:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:13:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:13:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:13:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:13:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:13:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:13:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:13:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:13:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:13:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:13:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:13:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:13:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:13:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:13:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:13:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:13:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:13:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:13:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:13:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:13:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:13:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:13:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:13:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:13:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:13:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:13:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:13:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:13:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:13:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:13:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:13:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:13:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:13:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:13:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:13:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:13:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:13:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:13:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:13:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:13:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:13:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:13:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:13:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:13:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:13:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:13:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:13:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:13:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:13:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:13:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:13:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:13:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:13:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:13:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:13:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:13:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:13:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:13:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:13:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:13:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:13:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:13:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:13:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:13:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:13:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:13:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:13:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:13:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:13:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:13:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:13:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:13:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:13:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:14:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:14:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:14:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:14:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:14:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:14:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:14:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:14:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:14:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:14:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:14:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:14:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:14:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:14:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:14:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:14:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:14:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:14:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:14:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:14:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:14:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:14:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:14:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:14:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:14:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:14:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:14:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:14:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:14:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:14:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:14:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:14:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:14:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:14:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:14:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:14:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:14:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:14:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:14:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:14:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:14:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:14:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:14:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:14:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:14:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:14:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:14:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:14:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:14:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:14:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:14:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:14:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:14:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:14:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:14:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:14:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:14:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:14:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:14:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:14:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:14:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:14:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:14:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:14:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:14:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:14:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:14:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:14:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:14:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:14:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:14:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:14:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:14:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:14:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:15:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:15:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:15:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:15:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:15:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:15:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:15:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:15:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:15:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:15:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:15:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:15:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:15:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:15:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:15:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:15:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:15:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:15:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:15:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:15:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:15:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:15:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:15:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:15:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:15:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:15:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:15:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:15:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:15:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:15:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:15:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:15:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:15:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:15:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:15:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:15:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:15:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:15:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:15:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:15:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:15:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:15:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:15:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:15:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:15:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:15:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:15:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:15:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:15:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:15:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:15:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:15:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:15:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:15:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:15:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:15:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:15:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:15:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:15:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:15:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:15:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:15:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:15:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:15:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:15:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:15:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:15:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:15:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:15:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:15:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:15:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:15:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:15:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:15:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:16:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:16:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:16:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:16:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:16:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:16:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:16:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:16:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:16:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:16:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:16:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:16:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:16:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:16:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:16:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:16:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:16:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:16:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:16:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:16:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:16:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:16:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:16:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:16:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:16:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:16:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:16:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:16:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:16:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:16:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:16:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:16:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:16:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:16:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:16:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:16:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:16:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:16:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:16:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:16:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:16:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:16:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:16:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:16:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:16:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:16:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:16:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:16:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:16:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:16:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:16:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:16:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:16:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:16:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:16:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:16:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:16:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:16:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:16:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:16:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:16:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:16:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:16:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:16:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:16:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:16:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:16:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:16:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:16:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:16:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:16:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:16:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:16:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:17:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:17:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:17:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:17:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:17:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:17:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:17:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:17:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:17:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:17:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:17:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:17:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:17:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:17:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:17:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:17:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:17:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:17:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:17:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:17:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:17:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:17:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:17:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:17:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:17:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:17:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:17:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:17:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:17:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:17:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:17:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:17:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:17:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:17:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:17:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:17:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:17:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:17:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:17:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:17:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:17:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:17:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:17:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:17:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: server_error: server error: 502" 2025-09-22 01:17:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:17:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:17:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:17:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:17:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:17:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:17:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:17:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:17:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:17:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:17:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:17:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:17:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:17:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:17:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:17:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:17:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:17:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:17:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:17:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:17:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:17:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:17:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:17:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:17:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:17:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:18:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:18:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:18:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:18:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:18:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:18:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:18:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:18:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:18:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:18:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:18:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:18:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:18:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:18:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:18:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:18:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:18:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:18:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:18:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:18:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:18:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:18:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:18:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:18:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:18:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:18:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:18:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:18:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:18:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:18:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:18:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:18:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:18:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:18:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:18:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:18:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:18:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:18:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:18:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:18:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:18:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:18:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:18:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:18:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:18:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:18:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:18:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:18:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:18:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:18:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:18:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:18:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:18:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:18:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:18:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:18:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:18:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:18:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:18:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:18:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:18:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:18:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:18:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:18:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:18:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:18:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:18:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:18:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:18:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:18:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:18:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:18:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:18:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:18:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:19:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:19:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:19:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:19:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:19:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:19:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:19:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:19:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:19:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:19:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:19:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:19:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:19:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:19:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:19:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:19:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:19:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:19:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:19:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:19:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:19:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:19:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:19:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:19:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:19:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:19:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:19:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:19:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:19:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:19:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:19:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:19:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:19:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:19:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:19:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:19:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:19:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:19:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:19:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:19:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:19:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:19:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:19:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:19:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:19:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:19:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:19:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:19:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:19:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:19:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:19:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:19:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:19:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:19:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:19:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:19:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:19:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:19:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:19:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:19:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:19:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:20:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:20:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:20:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:20:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:20:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:20:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:20:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:20:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:20:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:20:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:20:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:20:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:20:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:20:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:20:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:20:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:20:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:20:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:20:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:20:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:20:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:20:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:20:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:20:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:20:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:20:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:20:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:20:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:20:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:20:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:20:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:20:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:20:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:20:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:20:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:20:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:20:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:20:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:20:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:20:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:20:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:20:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:20:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:20:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:20:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:20:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:20:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:20:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:20:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:20:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:20:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:20:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:20:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:20:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:20:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:20:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:20:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:20:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:20:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:20:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:20:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:20:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:20:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:20:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:20:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:20:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:20:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:20:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:20:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:21:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:21:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:21:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:21:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:21:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:21:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:21:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:21:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:21:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:21:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:21:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:21:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:21:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:21:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:21:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:21:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:21:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:21:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:21:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:21:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:21:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:21:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:21:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:21:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:21:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:21:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:21:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:21:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:21:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:21:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:21:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:21:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:21:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:21:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:21:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:21:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:21:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:21:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:21:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:21:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:21:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:21:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:21:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:21:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:21:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:21:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:21:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:21:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:21:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:21:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:21:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:21:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:21:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:21:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:21:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:21:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:21:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:21:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:21:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:21:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:21:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:21:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:21:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:21:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:21:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:21:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:21:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:21:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:21:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:21:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:21:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:21:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:21:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:21:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:22:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:22:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:22:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:22:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:22:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:22:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:22:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:22:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:22:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:22:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:22:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:22:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:22:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:22:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:22:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:22:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:22:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:22:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:22:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:22:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:22:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:22:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:22:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:22:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:22:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:22:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:22:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:22:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:22:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:22:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:22:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:22:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:22:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:22:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:22:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:22:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:22:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:22:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:22:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:22:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:22:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:22:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:22:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:22:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:22:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:22:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:22:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:22:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:22:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:22:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:22:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:22:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:22:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:22:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:22:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:22:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:22:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:22:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:22:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:22:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:22:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:22:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:22:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:22:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:22:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:22:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:22:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:22:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:22:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:22:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:22:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:22:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:22:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:23:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:23:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:23:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:23:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:23:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:23:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:23:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:23:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:23:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:23:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:23:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:23:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:23:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:23:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:23:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:23:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:23:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:23:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:23:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:23:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:23:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:23:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:23:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:23:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:23:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:23:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:23:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:23:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:23:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:23:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:23:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:23:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:23:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:23:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:23:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:23:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:23:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:23:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:23:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:23:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:23:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:23:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:23:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:23:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:23:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:23:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:23:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:23:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:23:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:23:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:23:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:23:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:23:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:23:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:23:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:23:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:23:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:23:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:23:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:23:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:23:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:23:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:23:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:23:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:23:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:23:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:23:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:23:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:24:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:24:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:24:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:24:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:24:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:24:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:24:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:24:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:24:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:24:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:24:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:24:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:24:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:24:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:24:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:24:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:24:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:24:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:24:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:24:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:24:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:24:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:24:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:24:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:24:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:24:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:24:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:24:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:24:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:24:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:24:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:24:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:24:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:24:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:24:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:24:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:24:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:24:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:24:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:24:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:24:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:24:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:24:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:24:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:24:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:24:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:24:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:24:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:24:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:24:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:24:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:24:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:24:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:24:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:24:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:24:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:24:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:24:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:24:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:24:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:24:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:24:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:24:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:24:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:24:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:24:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:24:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:24:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:25:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:25:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:25:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:25:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:25:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:25:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:25:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:25:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:25:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:25:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:25:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:25:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:25:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:25:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:25:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:25:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:25:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:25:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:25:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:25:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:25:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:25:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:25:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:25:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:25:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:25:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:25:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:25:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:25:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:25:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:25:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:25:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:25:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:25:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:25:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:25:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: server_error: server error: 502" 2025-09-22 01:25:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:25:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:25:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:25:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:25:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:25:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:25:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:25:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:25:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:25:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:25:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:25:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:25:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:25:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:25:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:25:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:25:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:25:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:25:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:25:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:25:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:25:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:25:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:25:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:25:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:25:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:25:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:25:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:25:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:25:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:25:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:25:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:25:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:25:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:25:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:25:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:25:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:25:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:26:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:26:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:26:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:26:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:26:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:26:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:26:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:26:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:26:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:26:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:26:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:26:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:26:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:26:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:26:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:26:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:26:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:26:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:26:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:26:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:26:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:26:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:26:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:26:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:26:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:26:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:26:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:26:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:26:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:26:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:26:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:26:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:26:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:26:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:26:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:26:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:26:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:26:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:26:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:26:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:26:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:26:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:26:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:26:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:26:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:26:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:26:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:26:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:26:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:26:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:26:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:26:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:26:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:26:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:26:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:26:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:26:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:26:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:26:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:26:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:26:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:26:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:26:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:26:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:26:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:26:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:26:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:26:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:26:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:27:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:27:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:27:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:27:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:27:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:27:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:27:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:27:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:27:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:27:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:27:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:27:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:27:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:27:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:27:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:27:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:27:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:27:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:27:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:27:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:27:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:27:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:27:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:27:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:27:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:27:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:27:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:27:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:27:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:27:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:27:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:27:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:27:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:27:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:27:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:27:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:27:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:27:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:27:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:27:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:27:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:27:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:27:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:27:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:27:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:27:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:27:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:27:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:27:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:27:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:27:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:27:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:27:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:27:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:27:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:27:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:27:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:27:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:27:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:27:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:27:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:27:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:28:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:28:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:28:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:28:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:28:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:28:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:28:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:28:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:28:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:28:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:28:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:28:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:28:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:28:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:28:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:28:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:28:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:28:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:28:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:28:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:28:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:28:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:28:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:28:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:28:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:28:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:28:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:28:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:28:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:28:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:28:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:28:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:28:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:28:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:28:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:28:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:28:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:28:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:28:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:28:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:28:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:28:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:28:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:28:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:28:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:28:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:28:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:28:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:28:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:28:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:28:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:28:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:28:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:28:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:28:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:28:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:28:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:28:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:28:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:28:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:28:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:28:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:28:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:28:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:28:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:28:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:28:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:28:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:28:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:28:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:28:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:28:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:28:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:29:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:29:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:29:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:29:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:29:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:29:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:29:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:29:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:29:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:29:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:29:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:29:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:29:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:29:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:29:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:29:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:29:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:29:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:29:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:29:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:29:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:29:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:29:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:29:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:29:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:29:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:29:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:29:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:29:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:29:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:29:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:29:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:29:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:29:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:29:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:29:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:29:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:29:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:29:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:29:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:29:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:29:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:29:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:29:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:29:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:29:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:29:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:29:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:29:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:29:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:29:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:29:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:29:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:29:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:29:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:29:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:29:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:29:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:29:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:29:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:29:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:29:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:29:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:29:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:29:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:29:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:29:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:29:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:30:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:30:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:30:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:30:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:30:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:30:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:30:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:30:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:30:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:30:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:30:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:30:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:30:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:30:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:30:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:30:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:30:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:30:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:30:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:30:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:30:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:30:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:30:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:30:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:30:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:30:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:30:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:30:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:30:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:30:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:30:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:30:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:30:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:30:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:30:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:30:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:30:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:30:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:30:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:30:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:30:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:30:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:30:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:30:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:30:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:30:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:30:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:30:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:30:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:30:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:30:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:30:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:30:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:30:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:30:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:30:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:30:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:30:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:30:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:30:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:30:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:30:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:30:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:30:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:30:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:30:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:30:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:30:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:30:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:31:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:31:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:31:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:31:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:31:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:31:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:31:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:31:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:31:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:31:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:31:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:31:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:31:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:31:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:31:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:31:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:31:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:31:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:31:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:31:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:31:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:31:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:31:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:31:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: server_error: server error: 502" 2025-09-22 01:31:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:31:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:31:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:31:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:31:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:31:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:31:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:31:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:31:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:31:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:31:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:31:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:31:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:31:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:31:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:31:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:31:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:31:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:31:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:31:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:31:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:31:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:31:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:31:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:31:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:31:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:31:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:31:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:31:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:31:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:31:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:31:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:31:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:31:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:31:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:31:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:31:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:31:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:31:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:31:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:31:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:31:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:31:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:31:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:31:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:31:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:31:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:31:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:31:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:31:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:32:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:32:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:32:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:32:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:32:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:32:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:32:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:32:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:32:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:32:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:32:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:32:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:32:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:32:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:32:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:32:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:32:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:32:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:32:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:32:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:32:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:32:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:32:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:32:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:32:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:32:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:32:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:32:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:32:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:32:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:32:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:32:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:32:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:32:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:32:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:32:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:32:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:32:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:32:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:32:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:32:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:32:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:32:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:32:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:32:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:32:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:32:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:32:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:32:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:32:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:32:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:32:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:32:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:32:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:32:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:32:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:32:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:32:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:32:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:32:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:32:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:32:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:32:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:32:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:32:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:32:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:32:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:32:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:32:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:32:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:32:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:32:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:32:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:32:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:33:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:33:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:33:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:33:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:33:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:33:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:33:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:33:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:33:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:33:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:33:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:33:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:33:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:33:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:33:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:33:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:33:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:33:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:33:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:33:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:33:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:33:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:33:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:33:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:33:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:33:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:33:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:33:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:33:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:33:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:33:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:33:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:33:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:33:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:33:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:33:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:33:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:33:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:33:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:33:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:33:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:33:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:33:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:33:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:33:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:33:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:33:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:33:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:33:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:33:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:33:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:33:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:33:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:33:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:33:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:33:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:33:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:33:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:33:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:33:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:33:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:33:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:33:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:33:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:33:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:33:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:33:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:33:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:33:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:33:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:33:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:33:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:33:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:33:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:34:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:34:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:34:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:34:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:34:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:34:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:34:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:34:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:34:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:34:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:34:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:34:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:34:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:34:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:34:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:34:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:34:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:34:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:34:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:34:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:34:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:34:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:34:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:34:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:34:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:34:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:34:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:34:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:34:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:34:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:34:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:34:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:34:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:34:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:34:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:34:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:34:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:34:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:34:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:34:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:34:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:34:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:34:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:34:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:34:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:34:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:34:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:34:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:34:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:34:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:34:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:34:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:34:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:34:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:34:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:34:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:34:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:34:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:34:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:34:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:34:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:34:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:34:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:34:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:34:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:34:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:34:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:34:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:34:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:35:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:35:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:35:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:35:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:35:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:35:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:35:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:35:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:35:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:35:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:35:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:35:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:35:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:35:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:35:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:35:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:35:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:35:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:35:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:35:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:35:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:35:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:35:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:35:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:35:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:35:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:35:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:35:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:35:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:35:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:35:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:35:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:35:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:35:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:35:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:35:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:35:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:35:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:35:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:35:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:35:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:35:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:35:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:35:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:35:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:35:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:35:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:35:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:35:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:35:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:35:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:35:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:35:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:35:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:35:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:35:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:35:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:35:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:35:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:35:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:35:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:35:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:35:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:35:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:35:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:35:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:35:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:35:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:35:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:35:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:35:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:35:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:35:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:35:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:36:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:36:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:36:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:36:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:36:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:36:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:36:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:36:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:36:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:36:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:36:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:36:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:36:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:36:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:36:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:36:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:36:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:36:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:36:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:36:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:36:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:36:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:36:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:36:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:36:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:36:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:36:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:36:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:36:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:36:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:36:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:36:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:36:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:36:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:36:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:36:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:36:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:36:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:36:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:36:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:36:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:36:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:36:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:36:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:36:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:36:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:36:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:36:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:36:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:36:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:36:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:36:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:36:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:36:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:36:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:36:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:36:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:36:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:36:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:36:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:36:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:36:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:36:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:36:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:36:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:36:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:36:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:36:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:36:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:36:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:36:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:36:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:36:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:36:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:37:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:37:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:37:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:37:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:37:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:37:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:37:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:37:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:37:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:37:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:37:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:37:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:37:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:37:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:37:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:37:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:37:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:37:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:37:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:37:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:37:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:37:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:37:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:37:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:37:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:37:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:37:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:37:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:37:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:37:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:37:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:37:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:37:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:37:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:37:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:37:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:37:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:37:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:37:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:37:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:37:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:37:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:37:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:37:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:37:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:37:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:37:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:37:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:37:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:37:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:37:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:37:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:37:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:37:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:37:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:37:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:37:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:37:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:37:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:37:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:37:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:37:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:38:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:38:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:38:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:38:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:38:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:38:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:38:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:38:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:38:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:38:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:38:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:38:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:38:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:38:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:38:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:38:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:38:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:38:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:38:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:38:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:38:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:38:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:38:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:38:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:38:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:38:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:38:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:38:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:38:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:38:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:38:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:38:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:38:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:38:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:38:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:38:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:38:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:38:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:38:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:38:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:38:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:38:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:38:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:38:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:38:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:38:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:38:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:38:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:38:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:38:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:38:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:38:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:38:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:38:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:38:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:38:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:38:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:38:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:38:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:38:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:38:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:38:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:38:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:38:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:38:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:38:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:38:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:38:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:38:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:38:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:38:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:38:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:38:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:38:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:39:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:39:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:39:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:39:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:39:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:39:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:39:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:39:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:39:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:39:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:39:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:39:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:39:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:39:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:39:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:39:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:39:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:39:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:39:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:39:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:39:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:39:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:39:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:39:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:39:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:39:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:39:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:39:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:39:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:39:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:39:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:39:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:39:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:39:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:39:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:39:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:39:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:39:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:39:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:39:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:39:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:39:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:39:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:39:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:39:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:39:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:39:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:39:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:39:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:39:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:39:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:39:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:39:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:39:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:39:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:39:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:39:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:39:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:39:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:39:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:39:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:39:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:39:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:39:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:39:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:39:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:39:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:39:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:39:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:39:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:39:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:39:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:39:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:39:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:40:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:40:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:40:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:40:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:40:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:40:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:40:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:40:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:40:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:40:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:40:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:40:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:40:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:40:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:40:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:40:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:40:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:40:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:40:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:40:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:40:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:40:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:40:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:40:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:40:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:40:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:40:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:40:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:40:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:40:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:40:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:40:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:40:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:40:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:40:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:40:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:40:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:40:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:40:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:40:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:40:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:40:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:40:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:40:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:40:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:40:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:40:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:40:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:40:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:40:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:40:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:40:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:40:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:40:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:40:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:40:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:40:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:40:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:40:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:40:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:40:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:40:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:40:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:40:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:40:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:40:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:40:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:40:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:40:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:40:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:40:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:40:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:40:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:40:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:41:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:41:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:41:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:41:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:41:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:41:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:41:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:41:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:41:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:41:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:41:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:41:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:41:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:41:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:41:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:41:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:41:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:41:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:41:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:41:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:41:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:41:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:41:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:41:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:41:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:41:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:41:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:41:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:41:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:41:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:41:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:41:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:41:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:41:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:41:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:41:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:41:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:41:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:41:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:41:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:41:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:41:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:41:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:41:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:41:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:41:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:41:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:41:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:41:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:41:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:41:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:41:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:41:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:41:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:41:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:41:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:41:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:41:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:41:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:41:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:41:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:41:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:41:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:41:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:41:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:41:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:41:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:41:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:41:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:41:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:41:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:41:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:41:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:41:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:42:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:42:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:42:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:42:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:42:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:42:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:42:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:42:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:42:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:42:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:42:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:42:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:42:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:42:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:42:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:42:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:42:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:42:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:42:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:42:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:42:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:42:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:42:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:42:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:42:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:42:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:42:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:42:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:42:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:42:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:42:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:42:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:42:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:42:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:42:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:42:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:42:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:42:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:42:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:42:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:42:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:42:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:42:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:42:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:42:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:42:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:42:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:42:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:42:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:42:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:42:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:42:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:42:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:42:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:42:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:42:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:42:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:42:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:42:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:42:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:42:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:42:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:42:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:42:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:42:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:42:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:42:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:42:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:42:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:42:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:42:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:42:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:42:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:42:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:43:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:43:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:43:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:43:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:43:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:43:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:43:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:43:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:43:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:43:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:43:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:43:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:43:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:43:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:43:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:43:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:43:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:43:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:43:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:43:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:43:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:43:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:43:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:43:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:43:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:43:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:43:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:43:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:43:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:43:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:43:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:43:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:43:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:43:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:43:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:43:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:43:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:43:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:43:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:43:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:43:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:43:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:43:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:43:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:43:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:43:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:43:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:43:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:43:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:43:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:43:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:43:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:43:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:43:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:43:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:43:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:43:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:43:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:43:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:43:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:43:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:43:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:43:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:43:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:43:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:43:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:43:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:43:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:43:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:43:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:43:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:43:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:43:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:44:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:44:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:44:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:44:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:44:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:44:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:44:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:44:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:44:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:44:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:44:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:44:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:44:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:44:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:44:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:44:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:44:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:44:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:44:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:44:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:44:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:44:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:44:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:44:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:44:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:44:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:44:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:44:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:44:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:44:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:44:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:44:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:44:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:44:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:44:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:44:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:44:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:44:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:44:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:44:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:44:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:44:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:44:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:44:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:44:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:44:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:44:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:44:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:44:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:44:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:44:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:44:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:44:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:44:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:44:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:44:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:44:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:44:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:44:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:44:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:44:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:44:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:44:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:44:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:44:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:44:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:44:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:44:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:45:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:45:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:45:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:45:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:45:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:45:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:45:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:45:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:45:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:45:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:45:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:45:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:45:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:45:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:45:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:45:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:45:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:45:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:45:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:45:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:45:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:45:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:45:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:45:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:45:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:45:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:45:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:45:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:45:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:45:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:45:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:45:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:45:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:45:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:45:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:45:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:45:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:45:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:45:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:45:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:45:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:45:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:45:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:45:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:45:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:45:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:45:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:45:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:45:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:45:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:45:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:45:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:45:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:45:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:45:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:45:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:45:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:45:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:45:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:45:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:45:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:45:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:45:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:45:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:45:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:45:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:45:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:45:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:46:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:46:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:46:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:46:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:46:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:46:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:46:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:46:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:46:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:46:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:46:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:46:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:46:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:46:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:46:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:46:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:46:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:46:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:46:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:46:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:46:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:46:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:46:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:46:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:46:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:46:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:46:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:46:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:46:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:46:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:46:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:46:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:46:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:46:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:46:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:46:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:46:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:46:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:46:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:46:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:46:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:46:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:46:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:46:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:46:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:46:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:46:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:46:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:46:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:46:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:46:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:46:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:46:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:46:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:46:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:46:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:46:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:46:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:46:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:46:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:46:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:46:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:46:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:46:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:46:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:46:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:46:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:46:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:46:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:47:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:47:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:47:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:47:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:47:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:47:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:47:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:47:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:47:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:47:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:47:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:47:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:47:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:47:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:47:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:47:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:47:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:47:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:47:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:47:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:47:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:47:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:47:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:47:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:47:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:47:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:47:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:47:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:47:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:47:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:47:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:47:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:47:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:47:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:47:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:47:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:47:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:47:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:47:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:47:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:47:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:47:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:47:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:47:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:47:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:47:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:47:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:47:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:47:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:47:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:47:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:47:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:47:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:47:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:47:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:47:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:47:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:47:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:47:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:47:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:47:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:47:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:47:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:47:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:47:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:47:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:47:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:47:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:47:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:47:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:47:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:47:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:47:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:48:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:48:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:48:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:48:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:48:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:48:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:48:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:48:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:48:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:48:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:48:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:48:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:48:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:48:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:48:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:48:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:48:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:48:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:48:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:48:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:48:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:48:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:48:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:48:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:48:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:48:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:48:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:48:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:48:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:48:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:48:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:48:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:48:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:48:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:48:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:48:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:48:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:48:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:48:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:48:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:48:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:48:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:48:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:48:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:48:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:48:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:48:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:48:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:48:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:48:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:48:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:48:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:48:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:48:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:48:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:48:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:48:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:48:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:48:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:48:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:48:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:48:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:48:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:48:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:48:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:48:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:48:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:48:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:48:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:48:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:48:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:48:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:48:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:48:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:49:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:49:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:49:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:49:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:49:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:49:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:49:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:49:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:49:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:49:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:49:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:49:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:49:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:49:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:49:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:49:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:49:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:49:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:49:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:49:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:49:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:49:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:49:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:49:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:49:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:49:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:49:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:49:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:49:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:49:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:49:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:49:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:49:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:49:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:49:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:49:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:49:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:49:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:49:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:49:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:49:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:49:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:49:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:49:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:49:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:49:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:49:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:49:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:49:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:49:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:49:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:49:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:49:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:49:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:49:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:49:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:49:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:49:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:49:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:49:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:49:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:49:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:49:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:49:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:49:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:49:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:49:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:49:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:50:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:50:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:50:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:50:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:50:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:50:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:50:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:50:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:50:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:50:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:50:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:50:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:50:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:50:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:50:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:50:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:50:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:50:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:50:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:50:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:50:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:50:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:50:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:50:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:50:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:50:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:50:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:50:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:50:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:50:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:50:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:50:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:50:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:50:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:50:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:50:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:50:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:50:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:50:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:50:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:50:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:50:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:50:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:50:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:50:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:50:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:50:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:50:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:50:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:50:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:50:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:50:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:50:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:50:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:50:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:50:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:50:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:50:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:50:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:50:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:50:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:50:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:50:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:50:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:50:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:50:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:50:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:50:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:50:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:51:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:51:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:51:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:51:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:51:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:51:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:51:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:51:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:51:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:51:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:51:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:51:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:51:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:51:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:51:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:51:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:51:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:51:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:51:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:51:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:51:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:51:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:51:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:51:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:51:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:51:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:51:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:51:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:51:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:51:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:51:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:51:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:51:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:51:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:51:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:51:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:51:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:51:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:51:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:51:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:51:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:51:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:51:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:51:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:51:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:51:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:51:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:51:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:51:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:51:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:51:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:51:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:51:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:51:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:51:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:51:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:51:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:51:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:51:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:51:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:51:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:51:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:51:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:51:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:51:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:51:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:51:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:51:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:51:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:51:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:51:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:51:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:51:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:52:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:52:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:52:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:52:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:52:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:52:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:52:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:52:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:52:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:52:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:52:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:52:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:52:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:52:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:52:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:52:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:52:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:52:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:52:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:52:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:52:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:52:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:52:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:52:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:52:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:52:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:52:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:52:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:52:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:52:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:52:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:52:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:52:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:52:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:52:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:52:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:52:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:52:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:52:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:52:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:52:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:52:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:52:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:52:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:52:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:52:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:52:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:52:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:52:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:52:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:52:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:52:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:52:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:52:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:52:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:52:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:52:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:52:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:52:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:52:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:52:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:52:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:52:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:52:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:52:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:52:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:52:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:52:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:52:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:52:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:52:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:52:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:52:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:52:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:53:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:53:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:53:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:53:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:53:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:53:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:53:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:53:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:53:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:53:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:53:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:53:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:53:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:53:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:53:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:53:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:53:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:53:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:53:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:53:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:53:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:53:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:53:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:53:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:53:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:53:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:53:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:53:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:53:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:53:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:53:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:53:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:53:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:53:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:53:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:53:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:53:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:53:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:53:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:53:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:53:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:53:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:53:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:53:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:53:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:53:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:53:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:53:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:53:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:53:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:53:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:53:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:53:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:53:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:53:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:53:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:53:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:53:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:53:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:53:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:53:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:53:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:53:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:54:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:54:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:54:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:54:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:54:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:54:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:54:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:54:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:54:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:54:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:54:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:54:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:54:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:54:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:54:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:54:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:54:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:54:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:54:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:54:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:54:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:54:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:54:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:54:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:54:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:54:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:54:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:54:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:54:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:54:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:54:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:54:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:54:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:54:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:54:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:54:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:54:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:54:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:54:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:54:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:54:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:54:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:54:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:54:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:54:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:54:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:54:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:54:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:54:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:54:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:54:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:54:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:54:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:54:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:54:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:54:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:54:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:54:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:54:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:54:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:54:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:54:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:54:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:54:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:54:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:54:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:54:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:54:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:54:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:54:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:54:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:54:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:54:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:55:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:55:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:55:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:55:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:55:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:55:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:55:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:55:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:55:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:55:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:55:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:55:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:55:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:55:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:55:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:55:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:55:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:55:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:55:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:55:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:55:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:55:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:55:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:55:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:55:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:55:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:55:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:55:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:55:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:55:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:55:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:55:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:55:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:55:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:55:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:55:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:55:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:55:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:55:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:55:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:55:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:55:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:55:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:55:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:55:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:55:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:55:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:55:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:55:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:55:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:55:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:55:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:55:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:55:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:55:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:55:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:55:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:55:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:55:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:55:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:55:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:55:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:55:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:55:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:55:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:55:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:55:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:55:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:56:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:56:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:56:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:56:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:56:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:56:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:56:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:56:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:56:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:56:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:56:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:56:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:56:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:56:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:56:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:56:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:56:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:56:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:56:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:56:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:56:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:56:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:56:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:56:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:56:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:56:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:56:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:56:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:56:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:56:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:56:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:56:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:56:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:56:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:56:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:56:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:56:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:56:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:56:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:56:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:56:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:56:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:56:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:56:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:56:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:56:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:56:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:56:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:56:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:56:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:56:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:56:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:56:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:56:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:56:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:56:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:56:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:56:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:56:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:56:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:56:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:56:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:56:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:56:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:56:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:56:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:56:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:56:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:57:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:57:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:57:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:57:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:57:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:57:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:57:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:57:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:57:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:57:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:57:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:57:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:57:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:57:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:57:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:57:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:57:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:57:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:57:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:57:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:57:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:57:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:57:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:57:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:57:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:57:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:57:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:57:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:57:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:57:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:57:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:57:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:57:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:57:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:57:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:57:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:57:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:57:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:57:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:57:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:57:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:57:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:57:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:57:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:57:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:57:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:57:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:57:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:57:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:57:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:57:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:57:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:57:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:57:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:57:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:57:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:57:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:57:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:57:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:57:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:57:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:57:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:57:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:57:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:57:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:57:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:57:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:57:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:57:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:58:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:58:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:58:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:58:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:58:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:58:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:58:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:58:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:58:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:58:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:58:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:58:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:58:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:58:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-22 01:58:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:58:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-22 01:58:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-22 01:58:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-22 01:58:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-22 01:58:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-22 01:58:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-22 01:58:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:58:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-22 01:58:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-22 01:58:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-22 01:58:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:58:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-22 01:58:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-22 01:58:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-22 01:58:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-22 01:58:13 WRN Prometheus returned 401. Re-authenticating... 2025-09-22 01:58:13 INF Re-authenticating Prometheus client 2025-09-22 01:58:13 WRN Prometheus returned 401. Re-authenticating... 2025-09-22 01:58:13 WRN Prometheus returned 401. Re-authenticating... 2025-09-22 01:58:13 WRN Prometheus returned 401. Re-authenticating... 2025-09-22 01:58:13 WRN Prometheus returned 401. Re-authenticating... 2025-09-22 01:58:13 WRN Prometheus returned 401. Re-authenticating... 2025-09-22 07:58:59 INF Adding event: Scenario_benchmark-stop 2025-09-22 08:24:26 INF Stopping Lodemon 2025-09-22 08:24:26 INF Stopping monitoring service...