==================================================================================================== ============================ Pod describe lodemon-6fc57b65cd-l2kts ============================ ==================================================================================================== Name: lodemon-6fc57b65cd-l2kts Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Sun, 21 Sep 2025 04:21:26 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6fc57b65cd skaffold.dev/run-id=3fd9d821-4fd8-44c3-b4fe-5473414ef481 Annotations: Status: Running IP: 10.106.40.164 IPs: IP: 10.106.40.164 Controlled By: ReplicaSet/lodemon-6fc57b65cd Containers: lodemon: Container ID: containerd://34ce8ebf74e9fcdee3ead4b2430e1fee542e2fd6a11c787a6fcc7e8b0189b817 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Sun, 21 Sep 2025 04:21:26 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mkx72 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-mkx72: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-6fc57b65cd-l2kts ============================== ==================================================================================================== 2025-09-21 04:21:26 INF Starting lodemon server on port 8080 2025-09-21 04:21:26 INF Initializing clients... 2025-09-21 04:21:26 DBG Fetching ID token for accessing prometheus 2025-09-21 04:21:27 DBG Fetched ID token successfully 2025-09-21 04:21:27 INF Clients initialized successfully 2025-09-21 04:21:52 INF Configuring Lodemon 2025-09-21 04:21:52 INF Configuring monitoring units 2025-09-21 04:21:52 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-21 04:21:52 TRC Parsing Prometheus queries from config 2025-09-21 04:21:52 TRC Parsing component query: DS Java heap mem usage 2025-09-21 04:21:52 TRC Parsing component query: IDM Java heap mem usage 2025-09-21 04:21:52 TRC Parsing component query: AM Java heap mem usage 2025-09-21 04:21:52 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-21 04:21:52 TRC Parsing global query: 2025-09-21 04:21:52 TRC Parsing scoped query: DS Identity backend connections 2025-09-21 04:21:52 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-21 04:21:52 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-21 04:21:52 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-21 04:21:52 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-21 04:21:52 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-21 04:21:52 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-21 04:21:52 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-21 04:21:52 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-21 04:21:52 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-21 04:21:52 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-21 04:21:52 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-21 04:21:52 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-21 04:21:52 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-21 04:21:52 INF  - All monitoring units configured 2025-09-21 04:21:52 INF Configuring monitoring service 2025-09-21 04:21:52 INF  - Adding monitoring units to the service 2025-09-21 04:21:52 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-21 04:21:52 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-21 04:21:52 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-21 04:21:52 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-21 04:21:52 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-21 04:21:52 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-21 04:21:52 INF  - All monitoring units added to the service 2025-09-21 04:21:52 INF Starting Lodemon now 2025-09-21 04:21:52 INF Starting monitoring service... 2025-09-21 04:21:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:21:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:21:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:21:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:21:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:21:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:21:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:21:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:21:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:21:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:21:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:21:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:21:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:21:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:21:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:21:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:21:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:21:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:21:54 INF Adding event: Scenario_benchmark 2025-09-21 04:22:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:22:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:22:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:22:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:22:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:22:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:22:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:22:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:22:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:22:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:22:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:22:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:22:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:22:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:22:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:22:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:22:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:22:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:22:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:22:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:22:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:22:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:22:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:22:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:22:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:22:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:22:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:22:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:22:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:22:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:22:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:22:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:22:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:22:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:22:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:22:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:22:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:22:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:22:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:22:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:22:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:22:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:22:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:22:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:22:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:22:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:22:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:22:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:22:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:22:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:22:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:22:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:22:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:22:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:22:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:22:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:22:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:22:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:22:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:22:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:22:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:22:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:22:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:22:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:22:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:22:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:22:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:22:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:22:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:22:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:22:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:22:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:22:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:22:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:22:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:23:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:23:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:23:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:23:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:23:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:23:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:23:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:23:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:23:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:23:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:23:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:23:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:23:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:23:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:23:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:23:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:23:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:23:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:23:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:23:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:23:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:23:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:23:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:23:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:23:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:23:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:23:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:23:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:23:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:23:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:23:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:23:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:23:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:23:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:23:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:23:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:23:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:23:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:23:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:23:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:23:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:23:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:23:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:23:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:23:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:23:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:23:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:23:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:23:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:23:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:23:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:23:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:23:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:23:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:23:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:23:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:23:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:23:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:23:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:23:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:23:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:23:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:23:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:23:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:23:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:23:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:23:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:23:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:23:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:23:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:23:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:23:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:23:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:24:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:24:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:24:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:24:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:24:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:24:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:24:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:24:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:24:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:24:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:24:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:24:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:24:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:24:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:24:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:24:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:24:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:24:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:24:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:24:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:24:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:24:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:24:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:24:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:24:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:24:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:24:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:24:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:24:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:24:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:24:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:24:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:24:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:24:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:24:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:24:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:24:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:24:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:24:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:24:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:24:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:24:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:24:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:24:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:24:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:24:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:24:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:24:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:24:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:24:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:24:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:24:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:24:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:24:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:24:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:24:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:24:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:24:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:24:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:24:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:24:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:24:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:24:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:24:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:24:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:24:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:24:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:24:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:24:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:25:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:25:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:25:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:25:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:25:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:25:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:25:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:25:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:25:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:25:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:25:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:25:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:25:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:25:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:25:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:25:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:25:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:25:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:25:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:25:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:25:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:25:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:25:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:25:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:25:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:25:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:25:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:25:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:25:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:25:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:25:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:25:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:25:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:25:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:25:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:25:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:25:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:25:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:25:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:25:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:25:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:25:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:25:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:25:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:25:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:25:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:25:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:25:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:25:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:25:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:25:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:25:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:25:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:25:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:25:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:25:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:25:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:25:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:25:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:25:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:25:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:25:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:25:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:25:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:25:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:25:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:25:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:25:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:25:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:25:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:25:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:25:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:25:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:26:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:26:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:26:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:26:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:26:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:26:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:26:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:26:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:26:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:26:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:26:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:26:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:26:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:26:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:26:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:26:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:26:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:26:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:26:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:26:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:26:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:26:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:26:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:26:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:26:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:26:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:26:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:26:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:26:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:26:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:26:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:26:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:26:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:26:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:26:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:26:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:26:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:26:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:26:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:26:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:26:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:26:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:26:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:26:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:26:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:26:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:26:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:26:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:26:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:26:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:26:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:26:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:26:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:26:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:26:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:26:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:26:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:26:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:26:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:26:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:26:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:26:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:26:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:26:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:26:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:26:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:26:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:26:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:26:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:27:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:27:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:27:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:27:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:27:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:27:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:27:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:27:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:27:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:27:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:27:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:27:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:27:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:27:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:27:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:27:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:27:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:27:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:27:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:27:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:27:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:27:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:27:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:27:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:27:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:27:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:27:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:27:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:27:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:27:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:27:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:27:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:27:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:27:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:27:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:27:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:27:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:27:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:27:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:27:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:27:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:27:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:27:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:27:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:27:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:27:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:27:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:27:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:27:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:27:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:27:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:27:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:27:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:27:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:27:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:27:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:27:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:27:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:27:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:27:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:27:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:27:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:27:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:27:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:27:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:27:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:27:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:27:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:27:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:27:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:27:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:27:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:27:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:27:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:28:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:28:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:28:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:28:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:28:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:28:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:28:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:28:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:28:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:28:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:28:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:28:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:28:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:28:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:28:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:28:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:28:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:28:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:28:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:28:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:28:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:28:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:28:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:28:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:28:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:28:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:28:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:28:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:28:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:28:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:28:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:28:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:28:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:28:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:28:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:28:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:28:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:28:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:28:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:28:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:28:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:28:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:28:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:28:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:28:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:28:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:28:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:28:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:28:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:28:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:28:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:28:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:28:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:28:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:28:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:28:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:28:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:28:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:28:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:28:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:28:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:28:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:28:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:28:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:28:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:28:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:28:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:28:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:28:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:28:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:28:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:28:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:28:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:29:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:29:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:29:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:29:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:29:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:29:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:29:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:29:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:29:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:29:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:29:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:29:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:29:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:29:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:29:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:29:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:29:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:29:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:29:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:29:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:29:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:29:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:29:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:29:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:29:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:29:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:29:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:29:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:29:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:29:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:29:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:29:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:29:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:29:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:29:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:29:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:29:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:29:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:29:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:29:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:29:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:29:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:29:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:29:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:29:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:29:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:29:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:29:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:29:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:29:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:29:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:29:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:29:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:29:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:29:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:29:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:29:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:29:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:29:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:29:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:29:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:29:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:29:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:29:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:29:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:29:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:29:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:29:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:29:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:30:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:30:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:30:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:30:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:30:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:30:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:30:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:30:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:30:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:30:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:30:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:30:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:30:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:30:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:30:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:30:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:30:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:30:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:30:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:30:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:30:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:30:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:30:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:30:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:30:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:30:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:30:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:30:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:30:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:30:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:30:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:30:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:30:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:30:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:30:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:30:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:30:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:30:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:30:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:30:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:30:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:30:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:30:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:30:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:30:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:30:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:30:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:30:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:30:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:30:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:30:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:30:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:30:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:30:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:30:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:30:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:30:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:30:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:30:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:30:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:30:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:30:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:30:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:30:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:30:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:30:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:30:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:30:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:30:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:30:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:30:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:30:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:30:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:30:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:31:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:31:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:31:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:31:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:31:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:31:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:31:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:31:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:31:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:31:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:31:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:31:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:31:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:31:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:31:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:31:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:31:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:31:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:31:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:31:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:31:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:31:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:31:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:31:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:31:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:31:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:31:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:31:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:31:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:31:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:31:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:31:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:31:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:31:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:31:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:31:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:31:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:31:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:31:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:31:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:31:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:31:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:31:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:31:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:31:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:31:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:31:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:31:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:31:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:31:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:31:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:31:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:31:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:31:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:31:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:31:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:31:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:31:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:31:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:31:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:31:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:31:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:31:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:31:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:31:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:31:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:31:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:31:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:31:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:31:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:31:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:31:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:31:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:31:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:32:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:32:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:32:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:32:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:32:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:32:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:32:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:32:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:32:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:32:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:32:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:32:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:32:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:32:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:32:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:32:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:32:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:32:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:32:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:32:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:32:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:32:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:32:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:32:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:32:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:32:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:32:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:32:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:32:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:32:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:32:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:32:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:32:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:32:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:32:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:32:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:32:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:32:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:32:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:32:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:32:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:32:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:32:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:32:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:32:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:32:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:32:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:32:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:32:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:32:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:32:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:32:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:32:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:32:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:32:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:32:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:32:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:32:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:32:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:32:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:32:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:32:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:32:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:32:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:32:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:32:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:32:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:32:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:32:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:32:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:32:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:32:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:32:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:33:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:33:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:33:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:33:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:33:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:33:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:33:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:33:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:33:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:33:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:33:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:33:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:33:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:33:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:33:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:33:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:33:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:33:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:33:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:33:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:33:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:33:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:33:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:33:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:33:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:33:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:33:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:33:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:33:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:33:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:33:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:33:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:33:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:33:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:33:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:33:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:33:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:33:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:33:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:33:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:33:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:33:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:33:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:33:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:33:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:33:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:33:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:33:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:33:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:33:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:33:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:33:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:33:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:33:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:33:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:33:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:33:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:33:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:33:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:33:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:33:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:33:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:33:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:33:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:33:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:33:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:33:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:33:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:33:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:33:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:33:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:34:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:34:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:34:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:34:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:34:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:34:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:34:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:34:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:34:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:34:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:34:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:34:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:34:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:34:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:34:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:34:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:34:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:34:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:34:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:34:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:34:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:34:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:34:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:34:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:34:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:34:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:34:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:34:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:34:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:34:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:34:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:34:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:34:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:34:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:34:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:34:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:34:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:34:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:34:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:34:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:34:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:34:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:34:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:34:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:34:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:34:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:34:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:34:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:34:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:34:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:34:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:34:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:34:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:34:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:34:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:34:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:34:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:34:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:34:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:34:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:34:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:34:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:34:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:34:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:34:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:35:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:35:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:35:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:35:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:35:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:35:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:35:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:35:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:35:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:35:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:35:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:35:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:35:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:35:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:35:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:35:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:35:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:35:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:35:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:35:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:35:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:35:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:35:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:35:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:35:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:35:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:35:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:35:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:35:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:35:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:35:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:35:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:35:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:35:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:35:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:35:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:35:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:35:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:35:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:35:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:35:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:35:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:35:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:35:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:35:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:35:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:35:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:35:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:35:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:35:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:35:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:35:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:35:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:35:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:35:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:35:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:35:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:35:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:35:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:35:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:35:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:35:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:35:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:35:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:35:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:35:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:35:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:35:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:35:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:36:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:36:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:36:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:36:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:36:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:36:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:36:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:36:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:36:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:36:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:36:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:36:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:36:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:36:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:36:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:36:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:36:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:36:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:36:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:36:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:36:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:36:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:36:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:36:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:36:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:36:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:36:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:36:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:36:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:36:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:36:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:36:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:36:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:36:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:36:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:36:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:36:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:36:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:36:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:36:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:36:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:36:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:36:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:36:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:36:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:36:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:36:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:36:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:36:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:36:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:36:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:36:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:36:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:36:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:36:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:36:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:36:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:36:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:36:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:36:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:36:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:36:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:36:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:36:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:36:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:36:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:36:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:36:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:36:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:36:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:36:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:36:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:36:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:36:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:37:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:37:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:37:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:37:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:37:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:37:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:37:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:37:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:37:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:37:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:37:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:37:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:37:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:37:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:37:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:37:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:37:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:37:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:37:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:37:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:37:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:37:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:37:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:37:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:37:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:37:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:37:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:37:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:37:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:37:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:37:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:37:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:37:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:37:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:37:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:37:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:37:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:37:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:37:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:37:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:37:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:37:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:37:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:37:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:37:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:37:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:37:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:37:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:37:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:37:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:37:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:37:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:37:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:37:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:37:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:37:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:37:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:37:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:37:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:37:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:37:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:37:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:37:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:37:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:37:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:37:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:37:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:37:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:37:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:37:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:37:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:37:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:37:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:37:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:38:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:38:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:38:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:38:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:38:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:38:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:38:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:38:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:38:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:38:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:38:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:38:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:38:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:38:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:38:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:38:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:38:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:38:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:38:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:38:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:38:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:38:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:38:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:38:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:38:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:38:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:38:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:38:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:38:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:38:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:38:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:38:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:38:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:38:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:38:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:38:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:38:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:38:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:38:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:38:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:38:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:38:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:38:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:38:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:38:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:38:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:38:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:38:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:38:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:38:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:38:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:38:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:38:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:38:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:38:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:38:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:38:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:38:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:38:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:38:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:38:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:38:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:38:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:38:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:38:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:38:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:38:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:38:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:38:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:38:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:38:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:38:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:38:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:39:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:39:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:39:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:39:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:39:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:39:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:39:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:39:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:39:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:39:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:39:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:39:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:39:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:39:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:39:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:39:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:39:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:39:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:39:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:39:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:39:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:39:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:39:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:39:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:39:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:39:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:39:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:39:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:39:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:39:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:39:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:39:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:39:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:39:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:39:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:39:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:39:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:39:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:39:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:39:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:39:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:39:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:39:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:39:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:39:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:39:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:39:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:39:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:39:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:39:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:39:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:39:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:39:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:39:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:39:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:39:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:39:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:39:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:39:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:39:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:39:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:39:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:39:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:39:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:39:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:39:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:39:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:39:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:39:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:40:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:40:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:40:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:40:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:40:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:40:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:40:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:40:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:40:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:40:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:40:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:40:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:40:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:40:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:40:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:40:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:40:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:40:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:40:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:40:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:40:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:40:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:40:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:40:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:40:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:40:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:40:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:40:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:40:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:40:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:40:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:40:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:40:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:40:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:40:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:40:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:40:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:40:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:40:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:40:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:40:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:40:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:40:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:40:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:40:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:40:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:40:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:40:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:40:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:40:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:40:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:40:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:40:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:40:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:40:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:40:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:40:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:40:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:40:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:40:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:40:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:40:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:40:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:40:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:40:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:40:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:40:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:40:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:40:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:40:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:40:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:40:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:40:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:41:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:41:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:41:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:41:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:41:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:41:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:41:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:41:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:41:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:41:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:41:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:41:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:41:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:41:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:41:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:41:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:41:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:41:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:41:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:41:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:41:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:41:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:41:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:41:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:41:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:41:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:41:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:41:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:41:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:41:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:41:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:41:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:41:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:41:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:41:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:41:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:41:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:41:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:41:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:41:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:41:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:41:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:41:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:41:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:41:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:41:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:41:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:41:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:41:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:41:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:41:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:41:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:41:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:41:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:41:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:41:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:41:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:41:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:41:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:41:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:41:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:41:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:41:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:41:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:41:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:41:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:41:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:41:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:41:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:42:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:42:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:42:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:42:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:42:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:42:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:42:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:42:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:42:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:42:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:42:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:42:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:42:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:42:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:42:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:42:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:42:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:42:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:42:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:42:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:42:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:42:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:42:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:42:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:42:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:42:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:42:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:42:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:42:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:42:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:42:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:42:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:42:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:42:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:42:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:42:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:42:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:42:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:42:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:42:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:42:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:42:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:42:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:42:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:42:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:42:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:42:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:42:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:42:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:42:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:42:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:42:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:42:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:42:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:42:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:42:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:42:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:42:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:42:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:42:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:42:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:42:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:42:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:42:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:42:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:42:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:42:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:42:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:42:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:42:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:42:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:42:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:42:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:43:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:43:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:43:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:43:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:43:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:43:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:43:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:43:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:43:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:43:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:43:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:43:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:43:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:43:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:43:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:43:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:43:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:43:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:43:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:43:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:43:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:43:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:43:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:43:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:43:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:43:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:43:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:43:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:43:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:43:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:43:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:43:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:43:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:43:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:43:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:43:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:43:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:43:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:43:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:43:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:43:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:43:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:43:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:43:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:43:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:43:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:43:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:43:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:43:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:43:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:43:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:43:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:43:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:43:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:43:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:43:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:43:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:43:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:43:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:43:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:43:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:43:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:43:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:43:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:43:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:43:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:43:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:43:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:43:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:43:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:43:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:43:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:43:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:43:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:44:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:44:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:44:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:44:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:44:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:44:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:44:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:44:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:44:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:44:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:44:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:44:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:44:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:44:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:44:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:44:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:44:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:44:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:44:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:44:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:44:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:44:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:44:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:44:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:44:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:44:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:44:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:44:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:44:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:44:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:44:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:44:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:44:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:44:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:44:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:44:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:44:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:44:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:44:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:44:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:44:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:44:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:44:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:44:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:44:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:44:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:44:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:44:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:44:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:44:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:44:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:44:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:44:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:44:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:44:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:44:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:44:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:44:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:44:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:44:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:44:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:44:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:44:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:44:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:44:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:44:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:44:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:44:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:44:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:45:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:45:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:45:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:45:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:45:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:45:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:45:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:45:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:45:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:45:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:45:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:45:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:45:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:45:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:45:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:45:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:45:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:45:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:45:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:45:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:45:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:45:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:45:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:45:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:45:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:45:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:45:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:45:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:45:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:45:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:45:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:45:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:45:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:45:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:45:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:45:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:45:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:45:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:45:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:45:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:45:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:45:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:45:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:45:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:45:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:45:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:45:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:45:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:45:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:45:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:45:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:45:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:45:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:45:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:45:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:45:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:45:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:45:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:45:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:45:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:45:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:45:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:45:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:45:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:45:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:45:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:45:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:45:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:45:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:45:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:45:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:45:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:45:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:45:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:46:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:46:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:46:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:46:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:46:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:46:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:46:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:46:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:46:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:46:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:46:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:46:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:46:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:46:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:46:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:46:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:46:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:46:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:46:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:46:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:46:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:46:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:46:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:46:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:46:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:46:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:46:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:46:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:46:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:46:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:46:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:46:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:46:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:46:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:46:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:46:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:46:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:46:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:46:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:46:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:46:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:46:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:46:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:46:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:46:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:46:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:46:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:46:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:46:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:46:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:46:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:46:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:46:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:46:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:46:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:46:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:46:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:46:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:46:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:46:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:46:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:46:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:46:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:46:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:46:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:46:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:46:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:46:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:46:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:46:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:46:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:46:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:46:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:46:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:47:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:47:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:47:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:47:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:47:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:47:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:47:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:47:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:47:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:47:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:47:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:47:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:47:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:47:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:47:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:47:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:47:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:47:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:47:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:47:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:47:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:47:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:47:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:47:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:47:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:47:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:47:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:47:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:47:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:47:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:47:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:47:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:47:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:47:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:47:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:47:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:47:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:47:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:47:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:47:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:47:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:47:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:47:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:47:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:47:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:47:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:47:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:47:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:47:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:47:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:47:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:47:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:47:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:47:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:47:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:47:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:47:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:47:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:47:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:47:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:47:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:47:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:47:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:47:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:47:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:47:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:47:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:47:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:47:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:47:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:47:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:47:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:47:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:48:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:48:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:48:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:48:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:48:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:48:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:48:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:48:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:48:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:48:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:48:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:48:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:48:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:48:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:48:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:48:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:48:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:48:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:48:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:48:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:48:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:48:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:48:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:48:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:48:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:48:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:48:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:48:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:48:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:48:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:48:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:48:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:48:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:48:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:48:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:48:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:48:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:48:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:48:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:48:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:48:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:48:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:48:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:48:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:48:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:48:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:48:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:48:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:48:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:48:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:48:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:48:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:48:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:48:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:48:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:48:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:48:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:48:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:48:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:48:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:48:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:48:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:48:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:48:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:48:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:48:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:48:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:48:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:48:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:49:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:49:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:49:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:49:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:49:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:49:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:49:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:49:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:49:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:49:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:49:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:49:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:49:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:49:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:49:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:49:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:49:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:49:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:49:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:49:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:49:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:49:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:49:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:49:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:49:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:49:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:49:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:49:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:49:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:49:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:49:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:49:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:49:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:49:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:49:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:49:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:49:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:49:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:49:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:49:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:49:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:49:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:49:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:49:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:49:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:49:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:49:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:49:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:49:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:49:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:49:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:49:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:49:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:49:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:49:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:49:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:49:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:49:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:49:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:49:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:49:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:49:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:49:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:49:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:49:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:49:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:49:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:49:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:49:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:49:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:49:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:49:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:49:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:50:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:50:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:50:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:50:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:50:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:50:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:50:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:50:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:50:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:50:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:50:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:50:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:50:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:50:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:50:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:50:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:50:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:50:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:50:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:50:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:50:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:50:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:50:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:50:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:50:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:50:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:50:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:50:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:50:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:50:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:50:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:50:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:50:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:50:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:50:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:50:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:50:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:50:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:50:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:50:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:50:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:50:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:50:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:50:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:50:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:50:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:50:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:50:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:50:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:50:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:50:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:50:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:50:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:50:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:50:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:50:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:50:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:50:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:50:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:50:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:50:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:50:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:50:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:50:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:50:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:50:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:50:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:50:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:50:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:51:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:51:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:51:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:51:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:51:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:51:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:51:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:51:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:51:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:51:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:51:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:51:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:51:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:51:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:51:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:51:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:51:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:51:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:51:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:51:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:51:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:51:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:51:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:51:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:51:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:51:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:51:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:51:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:51:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:51:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:51:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:51:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:51:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:51:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:51:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:51:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:51:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:51:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:51:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:51:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:51:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:51:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:51:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:51:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:51:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:51:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:51:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:51:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:51:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:51:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:51:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:51:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:51:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:51:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:51:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:51:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:51:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:51:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:51:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:51:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:51:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:52:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:52:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:52:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:52:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:52:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:52:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:52:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:52:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:52:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:52:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:52:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:52:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:52:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:52:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:52:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:52:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:52:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:52:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:52:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:52:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:52:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:52:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:52:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:52:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:52:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:52:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:52:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:52:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:52:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:52:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:52:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:52:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:52:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:52:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:52:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:52:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:52:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:52:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:52:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:52:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:52:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:52:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:52:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:52:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:52:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:52:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:52:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:52:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:52:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:52:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:52:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:52:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:52:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:52:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:52:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:52:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:52:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:52:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:52:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:52:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:52:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:52:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:52:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:52:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:52:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:52:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:52:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:52:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:52:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:52:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:52:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:52:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:52:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:52:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:53:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:53:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:53:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:53:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:53:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:53:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:53:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:53:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:53:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:53:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:53:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:53:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:53:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:53:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:53:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:53:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:53:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:53:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:53:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:53:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:53:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:53:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:53:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:53:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:53:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:53:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:53:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:53:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:53:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:53:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:53:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:53:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:53:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:53:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:53:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:53:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:53:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:53:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:53:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:53:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:53:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:53:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:53:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:53:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:53:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:53:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:53:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:53:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:53:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:53:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:53:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:53:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:53:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:53:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:53:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:53:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:53:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:53:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:53:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:53:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:53:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:53:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:53:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:53:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:53:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:53:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:53:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:53:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:53:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:53:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:53:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:53:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:53:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:53:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:54:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:54:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:54:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:54:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:54:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:54:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:54:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:54:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:54:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:54:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:54:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:54:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:54:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:54:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:54:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:54:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:54:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:54:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:54:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:54:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:54:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:54:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:54:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:54:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:54:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:54:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:54:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:54:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:54:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:54:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:54:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:54:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:54:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:54:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:54:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:54:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:54:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:54:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:54:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:54:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:54:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:54:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:54:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:54:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:54:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:54:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:54:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:54:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:54:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:54:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:54:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:54:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:54:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:54:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:54:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:54:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:54:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:54:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:54:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:54:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:54:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:54:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:54:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:54:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:54:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:54:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:54:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:54:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:54:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:55:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:55:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:55:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:55:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:55:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:55:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:55:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:55:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:55:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:55:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:55:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:55:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:55:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:55:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:55:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:55:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:55:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:55:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:55:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:55:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:55:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:55:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:55:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:55:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:55:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:55:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:55:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:55:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:55:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:55:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:55:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:55:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:55:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:55:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:55:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:55:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:55:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:55:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:55:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:55:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:55:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:55:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:55:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:55:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:55:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:55:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:55:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:55:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:55:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:55:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:55:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:55:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:55:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:55:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:55:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:55:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:55:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:55:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:55:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:55:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:55:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:55:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:55:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:55:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:55:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:55:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:55:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:55:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:55:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:55:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:55:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:55:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:55:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:55:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:56:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:56:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:56:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:56:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:56:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:56:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:56:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:56:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:56:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:56:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:56:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:56:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:56:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:56:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:56:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:56:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:56:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:56:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:56:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:56:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:56:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:56:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:56:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:56:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:56:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:56:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:56:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:56:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:56:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:56:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:56:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:56:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:56:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:56:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:56:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:56:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:56:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:56:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:56:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:56:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:56:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:56:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:56:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:56:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:56:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:56:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:56:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:56:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:56:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:56:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:56:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:56:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:56:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:56:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:56:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:56:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:56:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:56:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:56:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:56:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:56:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:56:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:56:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:56:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:56:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:56:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:56:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:56:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:56:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:56:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:56:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:56:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:56:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:56:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:57:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:57:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:57:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:57:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:57:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:57:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:57:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:57:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:57:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:57:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:57:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:57:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:57:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:57:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:57:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:57:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:57:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:57:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:57:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:57:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:57:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:57:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:57:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:57:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:21 INF Adding event: Scenario_benchmark-stop 2025-09-21 04:57:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:57:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:57:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:57:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:57:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:57:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:57:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:57:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:57:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:57:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:57:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:57:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:57:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:57:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:57:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:57:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:57:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:57:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:57:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:57:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:57:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:57:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:57:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:57:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:57:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:57:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:57:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:57:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:57:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:57:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:57:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:57:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:57:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:57:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:57:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:57:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:57:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:57:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:57:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:57:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:57:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:57:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:57:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:57:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:57:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:57:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:57:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:57:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:57:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:57:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:57:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:58:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:58:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:58:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:58:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:58:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:58:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:58:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:58:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:58:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:58:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:58:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:58:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:58:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:58:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:58:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:58:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:58:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:58:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:58:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:58:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:58:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:58:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:58:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:58:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:58:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:58:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:58:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:58:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:58:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:58:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:58:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:58:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:58:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:58:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:58:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:58:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:58:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:58:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:58:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:58:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:58:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:58:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:58:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:58:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:58:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:58:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:58:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:58:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:58:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:58:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:58:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:58:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:58:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:58:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:58:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:58:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:58:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:58:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:58:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:58:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:58:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:58:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:58:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:58:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:58:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:58:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:58:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:58:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:58:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:58:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:58:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:58:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:58:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:59:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:59:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:59:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:59:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:59:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:59:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:59:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:59:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:59:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:59:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:59:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:59:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:59:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:59:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:59:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:59:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:59:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:59:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:59:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:59:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:59:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:59:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:59:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:59:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:59:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:59:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:59:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:59:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:59:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:59:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:59:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:59:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:59:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:59:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:59:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:59:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:59:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:59:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:59:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:59:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:59:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:59:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:59:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:59:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:59:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:59:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:59:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:59:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:59:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:59:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:59:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:59:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:59:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:59:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:59:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 04:59:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 04:59:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 04:59:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 04:59:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 04:59:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 04:59:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 04:59:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 04:59:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 04:59:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 04:59:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 04:59:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 04:59:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 04:59:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 04:59:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:00:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:00:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:00:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:00:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:00:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:00:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:00:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:00:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:00:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:00:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:00:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:00:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:00:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:00:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:00:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:00:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:00:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:00:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:00:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:00:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:00:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:00:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:00:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:00:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:00:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:00:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:00:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:00:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:00:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:00:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:00:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:00:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:00:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:00:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:00:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:00:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:00:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:00:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:00:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:00:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:00:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:00:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:00:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:00:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:00:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:00:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:00:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:00:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:00:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:00:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:00:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:00:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:00:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:00:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:00:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:00:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:00:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:00:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:00:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:00:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:00:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:00:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:00:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:00:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:00:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:00:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:00:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:00:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:00:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:00:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:00:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:00:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:00:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:00:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:01:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:01:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:01:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:01:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:01:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:01:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:01:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:01:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:01:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:01:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:01:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:01:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:01:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:01:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:01:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:01:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:01:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:01:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:01:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:01:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:01:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:01:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:01:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:01:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:01:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:01:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:01:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:01:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:01:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:01:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:01:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:01:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:01:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:01:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:01:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:01:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:01:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:01:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:01:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:01:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:01:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:01:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:01:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:01:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:01:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:01:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:01:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:01:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:01:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:01:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:01:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:01:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:01:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:01:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:01:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:01:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:01:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:01:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:01:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:01:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:01:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:01:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:01:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:01:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:01:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:01:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:01:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:01:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:01:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:01:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:01:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:01:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:01:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:02:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:02:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:02:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:02:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:02:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:02:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:02:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:02:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:02:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:02:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:02:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:02:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:02:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:02:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:02:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:02:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:02:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:02:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:02:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:02:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:02:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:02:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:02:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:02:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:02:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:02:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:02:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:02:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:02:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:02:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:02:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:02:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:02:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:02:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:02:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:02:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:02:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:02:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:02:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:02:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:02:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:02:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:02:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:02:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:02:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:02:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:02:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:02:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:02:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:02:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:02:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:02:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:02:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:02:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:02:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:02:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:02:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:02:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:02:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:02:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:02:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:02:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:02:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:02:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:02:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:02:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:02:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:02:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:02:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:03:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:03:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:03:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:03:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:03:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:03:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:03:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:03:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:03:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:03:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:03:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:03:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:03:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:03:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:03:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:03:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:03:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:03:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:03:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:03:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:03:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:03:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:03:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:03:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:03:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:03:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:03:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:03:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:03:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:03:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:03:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:03:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:03:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:03:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:03:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:03:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:03:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:03:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:03:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:03:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:03:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:03:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:03:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:03:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:03:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:03:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:03:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:03:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:03:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:03:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:03:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:03:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:03:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:03:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:03:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:03:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:03:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:03:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:03:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:03:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:03:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:03:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:03:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:03:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:03:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:03:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:03:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:03:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:03:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:03:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:03:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:03:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:03:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:04:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:04:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:04:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:04:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:04:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:04:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:04:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:04:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:04:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:04:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:04:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:04:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:04:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:04:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:04:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:04:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:04:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:04:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:04:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:04:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:04:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:04:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:04:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:04:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:04:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:04:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:04:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:04:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:04:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:04:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:04:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:04:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:04:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:04:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:04:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:04:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:04:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:04:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:04:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:04:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:04:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:04:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:04:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:04:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:04:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:04:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:04:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:04:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:04:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:04:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:04:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:04:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:04:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:04:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:04:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:04:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:04:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:04:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:04:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:04:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:04:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:04:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:04:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:04:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:04:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:04:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:04:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:04:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:04:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:05:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:05:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:05:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:05:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:05:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:05:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:05:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:05:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:05:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:05:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:05:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:05:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:05:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:05:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:05:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:05:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:05:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:05:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:05:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:05:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:05:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:05:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:05:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:05:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:05:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:05:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:05:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:05:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:05:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:05:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:05:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:05:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:05:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:05:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:05:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:05:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:05:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:05:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:05:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:05:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:05:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:05:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:05:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:05:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:05:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:05:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:05:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:05:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:05:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:05:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:05:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:05:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:05:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:05:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:05:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:05:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:05:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:05:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:05:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:05:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:05:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:05:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:05:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:05:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:05:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:05:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:05:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:05:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:05:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:05:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:05:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:05:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:05:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:06:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:06:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:06:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:06:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:06:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:06:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:06:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:06:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:06:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:06:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:06:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:06:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:06:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:06:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:06:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:06:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:06:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:06:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:06:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:06:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:06:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:06:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:06:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:06:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:06:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:06:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:06:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:06:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:06:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:06:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:06:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:06:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:06:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:06:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:06:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:06:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:06:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:06:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:06:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:06:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:06:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:06:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:06:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:06:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:06:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:06:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:06:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:06:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:06:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:06:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:06:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:06:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:06:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:06:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:06:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:06:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:06:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:06:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:06:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:06:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:06:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:06:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:06:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:06:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:06:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:06:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:06:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:06:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:06:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:06:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:06:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:06:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:06:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:06:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:07:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:07:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:07:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:07:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:07:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:07:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:07:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:07:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:07:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:07:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:07:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:07:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:07:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:07:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:07:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:07:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:07:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:07:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:07:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:07:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:07:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:07:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:07:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:07:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:07:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:07:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:07:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:07:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:07:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:07:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:07:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:07:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:07:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:07:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:07:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:07:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:07:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:07:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:07:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:07:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:07:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:07:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:07:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:07:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:07:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:07:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:07:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:07:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:07:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:07:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:07:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:07:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:07:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:07:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:07:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:07:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:07:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:07:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:07:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:07:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:07:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:07:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:07:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:07:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:07:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:07:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:07:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:07:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:08:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:08:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:08:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:08:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:08:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:08:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:08:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:08:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:08:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:08:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:08:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:08:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:08:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:08:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:08:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:08:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:08:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:08:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:08:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:08:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:08:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:08:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:08:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:08:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:08:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:08:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:08:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:08:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:08:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:08:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:08:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:08:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:08:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:08:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:08:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:08:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:08:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:08:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:08:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:08:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:08:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:08:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:08:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:08:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:08:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:08:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:08:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:08:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:08:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:08:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:08:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:08:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:08:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:08:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:08:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:08:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:08:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:08:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:08:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:08:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:08:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:08:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:08:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:09:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:09:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:09:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:09:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:09:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:09:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:09:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:09:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:09:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:09:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:09:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:09:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:09:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:09:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:09:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:09:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:09:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:09:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:09:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:09:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:09:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:09:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:09:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:09:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:09:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:09:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:09:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:09:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:09:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:09:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:09:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:09:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:09:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:09:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:09:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:09:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:09:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:09:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:09:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:09:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:09:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:09:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:09:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:09:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:09:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:09:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:09:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:09:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:09:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:09:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:09:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:09:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:09:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:09:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:09:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:09:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:09:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:09:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:09:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:09:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:09:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:09:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:09:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:09:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:09:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:09:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:09:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:09:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:09:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:09:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:09:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:09:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:09:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:10:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:10:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:10:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:10:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:10:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:10:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:10:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:10:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:10:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:10:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:10:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:10:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:10:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:10:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:10:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:10:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:10:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:10:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:10:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:10:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:10:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:10:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:10:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:10:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:10:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:10:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:10:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:10:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:10:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:10:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:10:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:10:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:10:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:10:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:10:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:10:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:10:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:10:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:10:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:10:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:10:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:10:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:10:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:10:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:10:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:10:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:10:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:10:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:10:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:10:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:10:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:10:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:10:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:10:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:10:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:10:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:10:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:10:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:10:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:10:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:10:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:10:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:10:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:10:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:10:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:10:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:10:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:10:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:10:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:11:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:11:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:11:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:11:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:11:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:11:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:11:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:11:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:11:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:11:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:11:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:11:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:11:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:11:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:11:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:11:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:11:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:11:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:11:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:11:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:11:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:11:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:11:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:11:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:11:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:11:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:11:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:11:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:11:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:11:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:11:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:11:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:11:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:11:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:11:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:11:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:11:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:11:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:11:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:11:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:11:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:11:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:11:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:11:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:11:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:11:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:11:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:11:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:11:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:11:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:11:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:11:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:11:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:11:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:11:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:11:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:11:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:11:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:11:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:11:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:11:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:11:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:11:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:11:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:11:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:11:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:11:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:11:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:11:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:11:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:11:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:11:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:11:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:11:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:12:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:12:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:12:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:12:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:12:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:12:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:12:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:12:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:12:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:12:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:12:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:12:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:12:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:12:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:12:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:12:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:12:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:12:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:12:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:12:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:12:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:12:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:12:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:12:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:12:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:12:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:12:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:12:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:12:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:12:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:12:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:12:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:12:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:12:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:12:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:12:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:12:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:12:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:12:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:12:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:12:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:12:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:12:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:12:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:12:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:12:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:12:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:12:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:12:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:12:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:12:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:12:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:12:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:12:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:12:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:12:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:12:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:12:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:12:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:12:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:12:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:12:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:12:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:12:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:12:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:12:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:12:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:12:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:12:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:12:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:12:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:12:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:12:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:13:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:13:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:13:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:13:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:13:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:13:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:13:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:13:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:13:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:13:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:13:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:13:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:13:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:13:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:13:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:13:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:13:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:13:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:13:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:13:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:13:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:13:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:13:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:13:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:13:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:13:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:13:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:13:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:13:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:13:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:13:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:13:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:13:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:13:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:13:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:13:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:13:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:13:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:13:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:13:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:13:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:13:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:13:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:13:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:13:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:13:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:13:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:13:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:13:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:13:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:13:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:13:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:13:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:13:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:13:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:13:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:13:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:13:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:13:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:13:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:13:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:13:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:13:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:13:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:13:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:13:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:13:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:13:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:14:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:14:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:14:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:14:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:14:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:14:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:14:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:14:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:14:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:14:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:14:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:14:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:14:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:14:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:14:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:14:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:14:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:14:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:14:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:14:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:14:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:14:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:14:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:14:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:14:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:14:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:14:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:14:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:14:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:14:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:14:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:14:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:14:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:14:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:14:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:14:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:14:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:14:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:14:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:14:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:14:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:14:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:14:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:14:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:14:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:14:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:14:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:14:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:14:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:14:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:14:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:14:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:14:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:14:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:14:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:14:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:14:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:14:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:14:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:14:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:14:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:14:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:14:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:14:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:14:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:14:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:14:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:14:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:15:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:15:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:15:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:15:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:15:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:15:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:15:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:15:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:15:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:15:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:15:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:15:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:15:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:15:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:15:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:15:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:15:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:15:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:15:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:15:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:15:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:15:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:15:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:15:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:15:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:15:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:15:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:15:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:15:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:15:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:15:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:15:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:15:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:15:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:15:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:15:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:15:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:15:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:15:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:15:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:15:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:15:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:15:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:15:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:15:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:15:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:15:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:15:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:15:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:15:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:15:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:15:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:15:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:15:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:15:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:15:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:15:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:15:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:15:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:15:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:15:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:15:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:15:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:15:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:15:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:15:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:15:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:15:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:15:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:16:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:16:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:16:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:16:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:16:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:16:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:16:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:16:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:16:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:16:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:16:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:16:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:16:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:16:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:16:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:16:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:16:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:16:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:16:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:16:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:16:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:16:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:16:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:16:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:16:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:16:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:16:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:16:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:16:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:16:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:16:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:16:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:16:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:16:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:16:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:16:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:16:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:16:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:16:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:16:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:16:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:16:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:16:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:16:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:16:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:16:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:16:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:16:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:16:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:16:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:16:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:16:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:16:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:16:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:16:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:16:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:16:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:16:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:16:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:16:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:16:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:16:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:16:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:16:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:16:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:16:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:16:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:16:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:16:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:16:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:16:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:16:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:16:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:17:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:17:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:17:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:17:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:17:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:17:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:17:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:17:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:17:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:17:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:17:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:17:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:17:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:17:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:17:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:17:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:17:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:17:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:17:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:17:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:17:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:17:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:17:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:17:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:17:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:17:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:17:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:17:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:17:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:17:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:17:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:17:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:17:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:17:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:17:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:17:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:17:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:17:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:17:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:17:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:17:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:17:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:17:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:17:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:17:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:17:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:17:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:17:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:17:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:17:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:17:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:17:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:17:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:17:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:17:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:17:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:17:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:17:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:17:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:17:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:17:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:17:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:17:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:17:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:17:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:17:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:17:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:17:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:18:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:18:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:18:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:18:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:18:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:18:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:18:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:18:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:18:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:18:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:18:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:18:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:18:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:18:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:18:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:18:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:18:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:18:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:18:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:18:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:18:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:18:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:18:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:18:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:18:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:18:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:18:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:18:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:18:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:18:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:18:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:18:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:18:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:18:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:18:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:18:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:18:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:18:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:18:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:18:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:18:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:18:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:18:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:18:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:18:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:18:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:18:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:18:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:18:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:18:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:18:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:18:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:18:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:18:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:18:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:18:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:18:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:18:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:18:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:18:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:18:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:18:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:18:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:18:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:18:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:18:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:18:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:18:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:19:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:19:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:19:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:19:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:19:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:19:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:19:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:19:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:19:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:19:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:19:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:19:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:19:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:19:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:19:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:19:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:19:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:19:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:19:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:19:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:19:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:19:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:19:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:19:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:19:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:19:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:19:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:19:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:19:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:19:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:19:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:19:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:19:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:19:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:19:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:19:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:19:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:19:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:19:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:19:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:19:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:19:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:19:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:19:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:19:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:19:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:19:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:19:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:19:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:19:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:19:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:19:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:19:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:19:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:19:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:19:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:19:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:19:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:19:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:19:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:19:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:19:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:19:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:19:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:19:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:19:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:19:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:19:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:20:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:20:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:20:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:20:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:20:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:20:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:20:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:20:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:20:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:20:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:20:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:20:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:20:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:20:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:20:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:20:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:20:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:20:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:20:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:20:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:20:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:20:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:20:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:20:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:20:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:20:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:20:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:20:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:20:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:20:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:20:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:20:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:20:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:20:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:20:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:20:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:20:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:20:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:20:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:20:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:20:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:20:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:20:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:20:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:20:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:20:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:20:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:20:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:20:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:20:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:20:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:20:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:20:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:20:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:20:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:20:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:20:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:20:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:20:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:20:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:20:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:20:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:20:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:20:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:20:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:20:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:20:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:20:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:20:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:20:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:20:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:20:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:20:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:20:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:21:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:21:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:21:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:21:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:21:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:21:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:21:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:21:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:21:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:21:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:21:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:21:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:21:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:21:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:21:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:21:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:21:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:21:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:21:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:21:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:21:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:21:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:21:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:21:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:21:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:21:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:21:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:21:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:21:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:21:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:21:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:21:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:21:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:21:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:21:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:21:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:21:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:21:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:21:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:21:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:21:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:21:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:21:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:21:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:21:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:21:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:21:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:21:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:21:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:21:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:21:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:21:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:21:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:21:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:21:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:21:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:21:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:21:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:21:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:21:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:21:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:21:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:21:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:21:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:21:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:21:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:21:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:21:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:21:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:22:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:22:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:22:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:22:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:22:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:22:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:22:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:22:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:22:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:22:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:22:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:22:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:22:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:22:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:22:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:22:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:22:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:22:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:22:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:22:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:22:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:22:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:22:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:22:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:22:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:22:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:22:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:22:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:22:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:22:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:22:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:22:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:22:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:22:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:22:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:22:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:22:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:22:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:22:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:22:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:22:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:22:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:22:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:22:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:22:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:22:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:22:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:22:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:22:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:22:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:22:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:22:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:22:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:22:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:22:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:22:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:22:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:22:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:22:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:22:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:22:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:22:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:22:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:22:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:22:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:22:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:22:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:22:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:22:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:22:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:22:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:22:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:22:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:22:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:23:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:23:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:23:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:23:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:23:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:23:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:23:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:23:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:23:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:23:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:23:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:23:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:23:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:23:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:23:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:23:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:23:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:23:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:23:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:23:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:23:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:23:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:23:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:23:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:23:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:23:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:23:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:23:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:23:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:23:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:23:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:23:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:23:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:23:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:23:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:23:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:23:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:23:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:23:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:23:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:23:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:23:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:23:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:23:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:23:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:23:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:23:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:23:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:23:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:23:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:23:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:23:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:23:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:23:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:23:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:23:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:23:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:23:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:23:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:23:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:23:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:23:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:23:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:23:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:23:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:23:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:23:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:23:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:23:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:23:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:23:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:23:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:23:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:24:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:24:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:24:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:24:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:24:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:24:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:24:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:24:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:24:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:24:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:24:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:24:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:24:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:24:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:24:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:24:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:24:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:24:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:24:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:24:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:24:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:24:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:24:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:24:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:24:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:24:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:24:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:24:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:24:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:24:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:24:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:24:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:24:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:24:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:24:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:24:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:24:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:24:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:24:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:24:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:24:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:24:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:24:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:24:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:24:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:24:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:24:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:24:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:24:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:24:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:24:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:24:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:24:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:24:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:24:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:24:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:24:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:24:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:24:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:24:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:24:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:24:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:24:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:24:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:24:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:24:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:24:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:24:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:24:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:25:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:25:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:25:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:25:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:25:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:25:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:25:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:25:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:25:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:25:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:25:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:25:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:25:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:25:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:25:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:25:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:25:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:25:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:25:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:25:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:25:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:25:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:25:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:25:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-21 05:25:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-21 05:25:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-21 05:25:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-21 05:25:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-21 05:25:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-21 05:25:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-21 05:25:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-21 05:25:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-21 05:25:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-21 05:25:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-21 05:25:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-21 05:25:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-21 05:25:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-21 05:25:30 INF Stopping Lodemon 2025-09-21 05:25:30 INF Stopping monitoring service... 2025-09-21 05:25:30 INF Lodemon stopped 2025-09-21 05:25:30 INF Generating reports 2025-09-21 05:25:30 TRC Result: &{DS Java heap mem usage [] [] } 2025-09-21 05:25:30 TRC Result: &{IDM Java heap mem usage [] [] } 2025-09-21 05:25:30 TRC Result: &{AM Java heap mem usage [] [] } 2025-09-21 05:25:30 TRC Result: &{Kubelet Volume Stats Inodes [] [] } 2025-09-21 05:25:30 TRC Result: &{container_memory_failcnt [] [] } 2025-09-21 05:25:30 TRC Result: &{DS Identity backend connections [] [] } 2025-09-21 05:25:30 INF Generating reports for 6 results into /tmp/lodemon/data/report.html