==================================================================================================== ============================ Pod describe lodemon-6bb875764d-4kbfk ============================ ==================================================================================================== Name: lodemon-6bb875764d-4kbfk Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Tue, 16 Sep 2025 05:33:59 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6bb875764d skaffold.dev/run-id=baf7baf8-4bd4-4d0b-94f2-e5f0f2e911bb Annotations: Status: Running IP: 10.106.40.158 IPs: IP: 10.106.40.158 Controlled By: ReplicaSet/lodemon-6bb875764d Containers: lodemon: Container ID: containerd://a9f5aa6c3f5d777cef25fec121d4ce524a0c4f1410ed56a3ad21c8c56bbc94c8 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:ea75a359346cdcf41c2b7c6e6a7fc13ee8f87f23c864140ee060af8927579d4e Port: 8080/TCP Host Port: 0/TCP State: Running Started: Tue, 16 Sep 2025 05:34:00 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-phmmd (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-phmmd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: ==================================================================================================== ============================== Pod logs lodemon-6bb875764d-4kbfk ============================== ==================================================================================================== 2025-09-16 05:34:00 INF Starting lodemon server on port 8080 2025-09-16 05:34:00 INF Initializing clients... 2025-09-16 05:34:00 DBG Fetching ID token for accessing prometheus 2025-09-16 05:34:00 DBG Fetched ID token successfully 2025-09-16 05:34:00 INF Clients initialized successfully 2025-09-16 05:34:35 INF Configuring Lodemon 2025-09-16 05:34:35 INF Configuring monitoring units 2025-09-16 05:34:35 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-16 05:34:35 TRC Parsing Prometheus queries from config 2025-09-16 05:34:35 TRC Parsing component query: DS Java heap mem usage 2025-09-16 05:34:35 TRC Parsing component query: IDM Java heap mem usage 2025-09-16 05:34:35 TRC Parsing component query: AM Java heap mem usage 2025-09-16 05:34:35 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-16 05:34:35 TRC Parsing global query: 2025-09-16 05:34:35 TRC Parsing scoped query: DS Identity backend connections 2025-09-16 05:34:35 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-16 05:34:35 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-16 05:34:35 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-16 05:34:35 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-16 05:34:35 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-16 05:34:35 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-16 05:34:35 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-16 05:34:35 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-16 05:34:35 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-16 05:34:35 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-16 05:34:35 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-16 05:34:35 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-16 05:34:35 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-16 05:34:35 INF  - All monitoring units configured 2025-09-16 05:34:35 INF Configuring monitoring service 2025-09-16 05:34:35 INF  - Adding monitoring units to the service 2025-09-16 05:34:35 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-16 05:34:35 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-16 05:34:35 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-16 05:34:35 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-16 05:34:35 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-16 05:34:35 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-16 05:34:35 INF  - All monitoring units added to the service 2025-09-16 05:34:35 INF Starting Lodemon now 2025-09-16 05:34:35 INF Starting monitoring service... 2025-09-16 05:34:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:34:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:34:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:34:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:34:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:34:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:34:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:34:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:34:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:34:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:34:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:34:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:34:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:41 INF Adding event: Scenario_benchmark 2025-09-16 05:34:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:34:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:34:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:34:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:34:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:34:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:34:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:34:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:34:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:34:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:34:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:34:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:34:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:34:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:34:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:34:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:34:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:34:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:34:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:34:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:34:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:34:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:34:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:34:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:34:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:34:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:34:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:35:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:35:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:35:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:35:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:35:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:35:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:35:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:35:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:35:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:35:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:35:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:35:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:35:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:35:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:35:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:35:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:35:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:35:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:35:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:35:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:35:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:35:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:35:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:35:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:35:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:35:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:35:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:35:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:35:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:35:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:35:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:35:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:35:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:35:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:35:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:35:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:35:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:35:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:35:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:35:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:35:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:35:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:35:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:35:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:35:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:35:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:35:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:35:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:35:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:35:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:35:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:35:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:35:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:35:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:35:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:35:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:35:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:35:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:35:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:35:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:35:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:35:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:35:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:35:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:35:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:35:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:35:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:35:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:35:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:35:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:35:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:35:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:35:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:36:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:36:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:36:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:36:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:36:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:36:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:36:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:36:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:36:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:36:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:36:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:36:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:36:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:36:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:36:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:36:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:36:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:36:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:36:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:36:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:36:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:36:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:36:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:36:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:36:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:36:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:36:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:36:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:36:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:36:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:36:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:36:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:36:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:36:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:36:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:36:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:36:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:36:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:36:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:36:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:36:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:36:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:36:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:36:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:36:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:36:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:36:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:36:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:36:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:36:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:36:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:36:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:36:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:36:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:36:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:36:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:36:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:36:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:36:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:36:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:36:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:36:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:36:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:36:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:36:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:36:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:36:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:36:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:37:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:37:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:37:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:37:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:37:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:37:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:37:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:37:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:37:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:37:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:37:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:37:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:37:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:37:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:37:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:37:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:37:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:37:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:37:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:37:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:37:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:37:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:37:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:37:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:37:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:37:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:37:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:37:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:37:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:37:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:37:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:37:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:37:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:37:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:37:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:37:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:37:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:37:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:37:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:37:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:37:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:37:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:37:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:37:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:37:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:37:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:37:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:37:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:37:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:37:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:37:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:37:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:37:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:37:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:37:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:37:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:37:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:37:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:37:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:37:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:37:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:37:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:37:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:37:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:37:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:37:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:37:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:37:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:37:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:38:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:38:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:38:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:38:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:38:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:38:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:38:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:38:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:38:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:38:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:38:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:38:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:38:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:38:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:38:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:38:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:38:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:38:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:38:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:38:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:38:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:38:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:38:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:38:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:38:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:38:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:38:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:38:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:38:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:38:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:38:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:38:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:38:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:38:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:38:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:38:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:38:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:38:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:38:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:38:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:38:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:38:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:38:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:38:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:38:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:38:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:38:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:38:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:38:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:38:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:38:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:38:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:38:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:38:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:38:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:38:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:38:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:38:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:38:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:38:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:38:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:38:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:38:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:38:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:38:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:38:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:38:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:38:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:38:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:38:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:38:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:38:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:38:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:38:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:39:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:39:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:39:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:39:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:39:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:39:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:39:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:39:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:39:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:39:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:39:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:39:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:39:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:39:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:39:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:39:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:39:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:39:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:39:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:39:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:39:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:39:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:39:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:39:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:39:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:39:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:39:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:39:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:39:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:39:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:39:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:39:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:39:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:39:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:39:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:39:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:39:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:39:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:39:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:39:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:39:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:39:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:39:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:39:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:39:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:39:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:39:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:39:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:39:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:39:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:39:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:39:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:39:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:39:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:39:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:39:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:39:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:39:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:39:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:39:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:39:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:39:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:39:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:39:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:39:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:39:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:39:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:39:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:39:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:39:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:39:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:39:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:39:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:39:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:40:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:40:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:40:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:40:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:40:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:40:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:40:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:40:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:40:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:40:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:40:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:40:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:40:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:40:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:40:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:40:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:40:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:40:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:40:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:40:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:40:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:40:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:40:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:40:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:40:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:40:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:40:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:40:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:40:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:40:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:40:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:40:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:40:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:40:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:40:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:40:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:40:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:40:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:40:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:40:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:40:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:40:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:40:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:40:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:40:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:40:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:40:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:40:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:40:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:40:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:40:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:40:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:40:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:40:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:40:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:40:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:40:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:40:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:40:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:40:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:40:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:40:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:40:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:40:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:40:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:40:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:40:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:40:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:40:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:40:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:40:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:40:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:40:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:41:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:41:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:41:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:41:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:41:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:41:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:41:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:41:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:41:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:41:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:41:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:41:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:41:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:41:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:41:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:41:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:41:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:41:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:41:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:41:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:41:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:41:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:41:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:41:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:41:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:41:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:41:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:41:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:41:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:41:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:41:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:41:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:41:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:41:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:41:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:41:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:41:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:41:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:41:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:41:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:41:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:41:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:41:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:41:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:41:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:41:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:41:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:41:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:41:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:41:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:41:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:41:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:41:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:41:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:41:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:41:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:41:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:41:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:41:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:41:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:41:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:41:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:41:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:42:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:42:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:42:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:42:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:42:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:42:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:42:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:42:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:42:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:42:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:42:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:42:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:42:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:42:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:42:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:42:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:42:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:42:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:42:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:42:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:42:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:42:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:42:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:42:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:42:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:42:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:42:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:42:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:42:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:42:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:42:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:42:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:42:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:42:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:42:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:42:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:42:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:42:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:42:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:42:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:42:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:42:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:42:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:42:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:42:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:42:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:42:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:42:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:42:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:42:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:42:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:42:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:42:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:42:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:42:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:42:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:42:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:42:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:42:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:42:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:42:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:42:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:42:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:42:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:42:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:42:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:42:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:42:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:42:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:42:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:42:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:42:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:42:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:42:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:43:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:43:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:43:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:43:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:43:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:43:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:43:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:43:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:43:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:43:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:43:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:43:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:43:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:43:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:43:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:43:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:43:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:43:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:43:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:43:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:43:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:43:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:43:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:43:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:43:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:43:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:43:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:43:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:43:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:43:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:43:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:43:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:43:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:43:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:43:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:43:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:43:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:43:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:43:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:43:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:43:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:43:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:43:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:43:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:43:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:43:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:43:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:43:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:43:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:43:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:43:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:43:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:43:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:43:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:43:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:43:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:43:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:43:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:43:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:43:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:43:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:43:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:43:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:43:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:43:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:43:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:43:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:43:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:43:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:43:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:43:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:43:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:43:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:43:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:44:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:44:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:44:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:44:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:44:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:44:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:44:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:44:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:44:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:44:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:44:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:44:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:44:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:44:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:44:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:44:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:44:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:44:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:44:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:44:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:44:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:44:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:44:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:44:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:44:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:44:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:44:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:44:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:44:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:44:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:44:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:44:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:44:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:44:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:44:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:44:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:44:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:44:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:44:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:44:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:44:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:44:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:44:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:44:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:44:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:44:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:44:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:44:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:44:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:44:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:44:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:44:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:44:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:44:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:44:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:44:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:44:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:44:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:44:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:44:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:44:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:44:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:44:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:44:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:44:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:44:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:44:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:44:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:44:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:44:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:44:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:44:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:44:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:45:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:45:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:45:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:45:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:45:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:45:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:45:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:45:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:45:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:45:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:45:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:45:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:45:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:45:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:45:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:45:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:45:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:45:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:45:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:45:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:45:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:45:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:45:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:45:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:45:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:45:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:45:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:45:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:45:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:45:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:45:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:45:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:45:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:45:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:45:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:45:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:45:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:45:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:45:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:45:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:45:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:45:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:45:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:45:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:45:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:45:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:45:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:45:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:45:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:45:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:45:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:45:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:45:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:45:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:45:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:45:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:45:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:45:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:45:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:45:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:45:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:45:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:45:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:45:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:45:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:45:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:45:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:45:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:46:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:46:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:46:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:46:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:46:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:46:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:46:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:46:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:46:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:46:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:46:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:46:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:46:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:46:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:46:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:46:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:46:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:46:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:46:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:46:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:46:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:46:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:46:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:46:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:46:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:46:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:46:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:46:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:46:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:46:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:46:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:46:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:46:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:46:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:46:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:46:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:46:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:46:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:46:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:46:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:46:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:46:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:46:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:46:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:46:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:46:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:46:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:46:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:46:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:46:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:46:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:46:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:46:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:46:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:46:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:46:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:46:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:46:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:46:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:46:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:46:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:46:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:46:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:46:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:46:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:46:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:46:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:46:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:46:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:46:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:46:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:46:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:46:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:46:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:47:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:47:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:47:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:47:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:47:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:47:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:47:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:47:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:47:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:47:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:47:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:47:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:47:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:47:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:47:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:47:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:47:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:47:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:47:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:47:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:47:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:47:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:47:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:47:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:47:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:47:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:47:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:47:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:47:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:47:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:47:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:47:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:47:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:47:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:47:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:47:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:47:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:47:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:47:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:47:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:47:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:47:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:47:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:47:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:47:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:47:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:47:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:47:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:47:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:47:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:47:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:47:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:47:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:47:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:47:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:47:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:47:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:47:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:47:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:47:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:47:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:47:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:47:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:47:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:47:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:47:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:47:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:47:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:47:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:48:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:48:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:48:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:48:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:48:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:48:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:48:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:48:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:48:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:48:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:48:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:48:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:48:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:48:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:48:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:48:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:48:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:48:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:48:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:48:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:48:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:48:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:48:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:48:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:48:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:48:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:48:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:48:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:48:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:48:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:48:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:48:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:48:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:48:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:48:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:48:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:48:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:48:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:48:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:48:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:48:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:48:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:48:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:48:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:48:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:48:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:48:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:48:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:48:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:48:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:48:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:48:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:48:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:48:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:48:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:48:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:48:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:48:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:48:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:48:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:48:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:48:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:48:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:48:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:48:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:48:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:48:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:48:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:48:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:48:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:48:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:48:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:48:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:49:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:49:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:49:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:49:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:49:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:49:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:49:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:49:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:49:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:49:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:49:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:49:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:49:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:49:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:49:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:49:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:49:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:49:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:49:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:49:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:49:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:49:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:49:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:49:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:49:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:49:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:49:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:49:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:49:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:49:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:49:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:49:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:49:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:49:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:49:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:49:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:49:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:49:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:49:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:49:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:49:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:49:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:49:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:49:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:49:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:49:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:49:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:49:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:49:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:49:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:49:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:49:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:49:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:49:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:49:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:49:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:49:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:49:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:49:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:49:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:49:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:49:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:49:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:49:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:49:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:49:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:49:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:49:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:50:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:50:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:50:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:50:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:50:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:50:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:50:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:50:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:50:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:50:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:50:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:50:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:50:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:50:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:50:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:50:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:50:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:50:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:50:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:50:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:50:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:50:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:50:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:50:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:50:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:50:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:50:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:50:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:50:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:50:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:50:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:50:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:50:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:50:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:50:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:50:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:50:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:50:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:50:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:50:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:50:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:50:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:50:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:50:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:50:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:50:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:50:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:50:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:50:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:50:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:50:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:50:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:50:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:50:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:50:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:50:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:50:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:50:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:50:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:50:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:50:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:50:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:50:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:50:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:50:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:50:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:50:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:50:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:51:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:51:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:51:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:51:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:51:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:51:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:51:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:51:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:51:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:51:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:51:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:51:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:51:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:51:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:51:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:51:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:51:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:51:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:51:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:51:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:51:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:51:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:51:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:51:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:51:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:51:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:51:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:51:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:51:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:51:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:51:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:51:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:51:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:51:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:51:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:51:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:51:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:51:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:51:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:51:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:51:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:51:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:51:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:51:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:51:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:51:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:51:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:51:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:51:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:51:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:51:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:51:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:51:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:51:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:51:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:51:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:51:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:51:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:51:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:51:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:51:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:51:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:51:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:51:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:51:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:51:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:51:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:51:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:52:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:52:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:52:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:52:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:52:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:52:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:52:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:52:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:52:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:52:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:52:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:52:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:52:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:52:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:52:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:52:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:52:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:52:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:52:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:52:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:52:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:52:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:52:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:52:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:52:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:52:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:52:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:52:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:52:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:52:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:52:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:52:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:52:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:52:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:52:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:52:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:52:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:52:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:52:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:52:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:52:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:52:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:52:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:52:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:52:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:52:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:52:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:52:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:52:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:52:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:52:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:52:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:52:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:52:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:52:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:52:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:52:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:52:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:52:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:52:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:52:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:52:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:52:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:52:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:52:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:52:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:52:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:52:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:52:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:53:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:53:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:53:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:53:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:53:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:53:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:53:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:53:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:53:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:53:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:53:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:53:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:53:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:53:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:53:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:53:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:53:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:53:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:53:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:53:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:53:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:53:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:53:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:53:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:53:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:53:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:53:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:53:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:53:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:53:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:53:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:53:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:53:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:53:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:53:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:53:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:53:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:53:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:53:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:53:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:53:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:53:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:53:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:53:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:53:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:53:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:53:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:53:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:53:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:53:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:53:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:53:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:53:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:53:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:53:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:53:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:53:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:53:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:53:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:53:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:53:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:53:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:53:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:53:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:53:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:53:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:53:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:53:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:53:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:53:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:53:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:53:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:53:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:54:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:54:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:54:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:54:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:54:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:54:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:54:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:54:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:54:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:54:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:54:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:54:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:54:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:54:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:54:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:54:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:54:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:54:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:54:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:54:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:54:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:54:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:54:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:54:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:54:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:54:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:54:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:54:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:54:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:54:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:54:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:54:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:54:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:54:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:54:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:54:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:54:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:54:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:54:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:54:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:54:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:54:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:54:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:54:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:54:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:54:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:54:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:54:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:54:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:54:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:54:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:54:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:54:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:54:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:54:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:54:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:54:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:54:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:54:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:54:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:54:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:54:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:54:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:54:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:54:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:54:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:54:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:54:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:54:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:54:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:54:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:54:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:54:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:54:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:55:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:55:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:55:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:55:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:55:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:55:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:55:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:55:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:55:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:55:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:55:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:55:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:55:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:55:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:55:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:55:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:55:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:55:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:55:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:55:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:55:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:55:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:55:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:55:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:55:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:55:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:55:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:55:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:55:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:55:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:55:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:55:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:55:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:55:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:55:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:55:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:55:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:55:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:55:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:55:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:55:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:55:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:55:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:55:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:55:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:55:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:55:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:55:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:55:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:55:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:55:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:55:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:55:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:55:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:55:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:55:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:55:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:55:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:55:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:55:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:55:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:55:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:55:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:55:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:55:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:55:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:55:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:55:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:55:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:56:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:56:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:56:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:56:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:56:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:56:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:56:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:56:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:56:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:56:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:56:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:56:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:56:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:56:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:56:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:56:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:56:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:56:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:56:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:56:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:56:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:56:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:56:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:56:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:56:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:56:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:56:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:56:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:56:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:56:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:56:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:56:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:56:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:56:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:56:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:56:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:56:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:56:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:56:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:56:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:56:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:56:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:56:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:56:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:56:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:56:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:56:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:56:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:56:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:56:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:56:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:56:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:56:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:56:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:56:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:56:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:56:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:56:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:56:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:56:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:56:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:56:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:56:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:56:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:56:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:56:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:56:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:56:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:56:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:56:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:56:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:56:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:56:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:57:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:57:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:57:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:57:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:57:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:57:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:57:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:57:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:57:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:57:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:57:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:57:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:57:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:57:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:57:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:57:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:57:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:57:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:57:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:57:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:57:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:57:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:57:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:57:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:57:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:57:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:57:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:57:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:57:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:57:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:57:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:57:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:57:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:57:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:57:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:57:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:57:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:57:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:57:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:57:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:57:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:57:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:57:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:57:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:57:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:57:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:57:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:57:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:57:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:57:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:57:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:57:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:57:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:57:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:57:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:57:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:57:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:57:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:57:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:57:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:57:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:57:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:57:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:57:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:57:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:57:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:57:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:57:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:57:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:58:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:58:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:58:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:58:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:58:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:58:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:58:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:58:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:58:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:58:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:58:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:58:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:58:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:58:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:58:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:58:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:58:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:58:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:58:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:58:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:58:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:58:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:58:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:58:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:58:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:58:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:58:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:58:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:58:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:58:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:58:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:58:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:58:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:58:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:58:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:58:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:58:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:58:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:58:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:58:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:58:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:58:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:58:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:58:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:58:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:58:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:58:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:58:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:58:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:58:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:58:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:58:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:58:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:58:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:58:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:58:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:58:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:58:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:58:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:58:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:58:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:58:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:58:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:58:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:58:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:58:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:58:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:58:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:58:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:58:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:58:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:58:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:58:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:59:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:59:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:59:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:59:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:59:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:59:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:59:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:59:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:59:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:59:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:59:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:59:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:59:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:59:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:59:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:59:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:59:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:59:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:59:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:59:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:59:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:59:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:59:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:59:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:59:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:59:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:59:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:59:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:59:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:59:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:59:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:59:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:59:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:59:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:59:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:59:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:59:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:59:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:59:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:59:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:59:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:59:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:59:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:59:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:59:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:59:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:59:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:59:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 05:59:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 05:59:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 05:59:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 05:59:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 05:59:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 05:59:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 05:59:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 05:59:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 05:59:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 05:59:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 05:59:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 05:59:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 05:59:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 05:59:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:00:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:00:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:00:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:00:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:00:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:00:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:00:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:00:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:00:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:00:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:00:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:00:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:00:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:00:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:00:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:00:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:00:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:00:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:00:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:00:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:00:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:00:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:00:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:00:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:00:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:00:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:00:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:00:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:00:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:00:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:00:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:00:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:00:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:00:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:00:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:00:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:00:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:00:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:00:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:00:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:00:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:00:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:00:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:00:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:00:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:00:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:00:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:00:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:00:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:00:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:00:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:00:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:00:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:00:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:00:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:00:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:00:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:00:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:00:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:00:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:00:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:00:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:00:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:00:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:00:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:00:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:00:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:00:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:00:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:01:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:01:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:01:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:01:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:01:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:01:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:01:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:01:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:01:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:01:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:01:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:01:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:01:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:01:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:01:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:01:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:01:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:01:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:01:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:01:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:01:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:01:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:01:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:01:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:01:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:01:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:01:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:01:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:01:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:01:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:01:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:01:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:01:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:01:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:01:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:01:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:01:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:01:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:01:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:01:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:01:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:01:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:01:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:01:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:01:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:01:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:01:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:01:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:01:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:01:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:01:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:01:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:01:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:01:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:01:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:01:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:01:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:01:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:01:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:01:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:01:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:01:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:01:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:01:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:01:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:01:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:01:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:01:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:01:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:01:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:01:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:01:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:01:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:02:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:02:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:02:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:02:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:02:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:02:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:02:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:02:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:02:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:02:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:02:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:02:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:02:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:02:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:02:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:02:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:02:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:02:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:02:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:02:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:02:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:02:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:02:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:02:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:02:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:02:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:02:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:02:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:02:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:02:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:02:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:02:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:02:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:02:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:02:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:02:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:02:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:02:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:02:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:02:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:02:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:02:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:02:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:02:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:02:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:02:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:02:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:02:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:02:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:02:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:02:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:02:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:02:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:02:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:02:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:02:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:02:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:02:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:02:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:02:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:02:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:02:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:02:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:02:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:02:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:02:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:02:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:02:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:03:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:03:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:03:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:03:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:03:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:03:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:03:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:03:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:03:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:03:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:03:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:03:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:03:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:03:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:03:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:03:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:03:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:03:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:03:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:03:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:03:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:03:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:03:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:03:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:03:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:03:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:03:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:03:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:03:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:03:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:03:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:03:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:03:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:03:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:03:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:03:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:03:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:03:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:03:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:03:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:03:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:03:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:03:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:03:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:03:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:03:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:03:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:03:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:03:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:03:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:03:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:03:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:03:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:03:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:03:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:03:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:03:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:03:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:03:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:03:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:03:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:03:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:03:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:03:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:03:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:03:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:03:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:03:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:03:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:04:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:04:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:04:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:04:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:04:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:04:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:04:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:04:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:04:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:04:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:04:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:04:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:04:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:04:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:04:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:04:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:04:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:04:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:04:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:04:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:04:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:04:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:04:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:04:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:04:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:04:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:04:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:04:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:04:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:04:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:04:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:04:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:04:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:04:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:04:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:04:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:04:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:04:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:04:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:04:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:04:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:04:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:04:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:04:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:04:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:04:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:04:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:04:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:04:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:04:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:04:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:04:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:04:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:04:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:04:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:04:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:04:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:04:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:04:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:04:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:04:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:04:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:04:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:04:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:04:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:04:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:04:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:04:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:04:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:04:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:04:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:04:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:04:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:05:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:05:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:05:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:05:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:05:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:05:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:05:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:05:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:05:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:05:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:05:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:05:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:05:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:05:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:05:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:05:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:05:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:05:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:05:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:05:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:05:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:05:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:05:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:05:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:05:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:05:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:05:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:05:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:05:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:05:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:05:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:05:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:05:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:05:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:05:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:05:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:05:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:05:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:05:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:05:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:05:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:05:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:05:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:05:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:05:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:05:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:05:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:05:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:05:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:05:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:05:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:05:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:05:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:05:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:05:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:05:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:05:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:05:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:05:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:05:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:05:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:05:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:05:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:05:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:05:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:05:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:05:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:05:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:05:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:06:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:06:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:06:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:06:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:06:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:06:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:06:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:06:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:06:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:06:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:06:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:06:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:06:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:06:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:06:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:06:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:06:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:06:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:06:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:06:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:06:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:06:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:06:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:06:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:06:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:06:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:06:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:06:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:06:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:06:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:06:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:06:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:06:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:06:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:06:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:06:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:06:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:06:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:06:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:06:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:06:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:06:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:06:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:06:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:06:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:06:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:06:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:06:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:06:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:06:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:06:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:06:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:06:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:06:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:06:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:06:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:06:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:06:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:06:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:06:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:06:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:06:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:06:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:06:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:06:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:06:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:06:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:06:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:06:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:06:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:06:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:06:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:06:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:07:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:07:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:07:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:07:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:07:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:07:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:07:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:07:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:07:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:07:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:07:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:07:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:07:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:07:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:07:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:07:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:07:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:07:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:07:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:07:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:07:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:07:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:07:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:07:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:07:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:07:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:07:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:07:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:07:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:07:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:07:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:07:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:07:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:07:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:07:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:07:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:07:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:07:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:07:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:07:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:07:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:07:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:07:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:07:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:07:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:07:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:07:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:07:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:07:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:07:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:07:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:07:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:07:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:07:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:07:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:07:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:07:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:07:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:07:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:07:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:07:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:07:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:07:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:07:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:07:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:07:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:07:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:07:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:07:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:08:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:08:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:08:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:08:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:08:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:08:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:08:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:08:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:08:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:08:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:08:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:08:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:08:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:08:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:08:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:08:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:08:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:08:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:08:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:08:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:08:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:08:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:08:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:08:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:08:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:08:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:08:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:08:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:08:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:08:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:08:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:08:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:08:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:08:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:08:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:08:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:08:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:08:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:08:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:08:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:08:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:08:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:08:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:08:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:08:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:08:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:08:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:08:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:08:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:08:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:08:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:08:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:08:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:08:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:08:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:08:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:08:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:08:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:08:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:08:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:08:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:08:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:08:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:08:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:08:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:08:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:08:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:08:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:08:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:08:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:08:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:08:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:08:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:08:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:09:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:09:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:09:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:09:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:09:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:09:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:09:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:09:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:09:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:09:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:09:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:09:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:09:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:09:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:09:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:09:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:09:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:09:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:09:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:09:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:09:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:09:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:09:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:09:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:09:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:09:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:09:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:09:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:09:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:09:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:09:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:09:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:09:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:09:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:09:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:09:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:09:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:09:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:09:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:09:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:09:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:09:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:09:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:09:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:09:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:09:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:09:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:09:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:09:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:09:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:09:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:09:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:09:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:09:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:09:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:09:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:09:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:09:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:09:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:09:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:09:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:09:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:09:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:09:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:09:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:09:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:09:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:09:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:09:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:09:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:09:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:09:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:09:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:09:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:10:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:10:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:10:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:10:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:10:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:10:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:10:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:10:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:10:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:10:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:10:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:10:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:10:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:10:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:10:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:10:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:10:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:10:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:10:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:10:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:10:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:10:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:10:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:10:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:10:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:10:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:10:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:10:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:10:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:10:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:10:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:10:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:10:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:10:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:10:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:10:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:10:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:10:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:10:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:10:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:10:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:10:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:10:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:10:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:10:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:10:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:10:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:10:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:10:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:10:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:10:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:10:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:10:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:10:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:10:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:10:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:10:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:10:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:10:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:10:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:10:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:10:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:10:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:10:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:10:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:10:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:10:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:10:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:10:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:10:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:10:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:10:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:10:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:10:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:11:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:11:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:11:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:11:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:11:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:11:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:11:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:11:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:11:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:11:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:11:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:11:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:11:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:11:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:11:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:11:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:11:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:11:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:11:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:11:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:11:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:11:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:11:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:11:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:11:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:11:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:11:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:11:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:11:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:11:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:11:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:11:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:11:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:11:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:11:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:11:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:11:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:11:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:11:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:11:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:11:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:11:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:11:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:11:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:11:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:11:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:11:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:11:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:11:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:11:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:11:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:11:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:11:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:11:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:11:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:11:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:11:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:11:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:11:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:11:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:11:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:11:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:11:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:11:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:11:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:11:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:11:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:11:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:11:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:11:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:11:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:11:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:11:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:11:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:12:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:12:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:12:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:12:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:12:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:12:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:12:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:12:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:12:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:12:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:12:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:12:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:12:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:12:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:12:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:12:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:12:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:12:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:12:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:12:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:12:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:12:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:12:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:12:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:12:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:12:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:12:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:12:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:12:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:12:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:12:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:12:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:12:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:12:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:12:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:12:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:12:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:12:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:12:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:12:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:12:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:12:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:12:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:12:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:12:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:12:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:12:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:12:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:12:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:12:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:12:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:12:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:12:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:12:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:12:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:12:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:12:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:12:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:12:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:12:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:12:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:12:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:12:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:12:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:12:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:12:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:12:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:12:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:12:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:12:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:12:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:12:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:12:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:12:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:13:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:13:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:13:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:13:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:13:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:13:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:13:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:13:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:13:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:13:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:13:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:13:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:13:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:13:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:13:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:13:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:13:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:13:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:13:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:13:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:13:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:13:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:13:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:13:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:13:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:13:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:13:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:13:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:13:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:13:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:13:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:13:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:13:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:13:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:13:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:13:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:13:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:13:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:13:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:13:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:13:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:13:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:13:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:13:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:13:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:13:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:13:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:13:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:13:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:13:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:13:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:13:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:13:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:13:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:13:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:13:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:13:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:13:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:13:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:13:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:13:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:13:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:13:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:13:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:13:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:13:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:13:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:13:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:13:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:13:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:13:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:13:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:13:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:14:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:14:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:14:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:14:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:14:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:14:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:14:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:14:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:14:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:14:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:14:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:14:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:14:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:14:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:14:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:14:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:14:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:14:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:14:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:14:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:14:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:14:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:14:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:14:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:14:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:14:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:14:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:14:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:14:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:14:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:14:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:14:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:14:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:14:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:14:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:14:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:14:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:14:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:14:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:14:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:14:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:14:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:14:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:14:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:14:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:14:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:14:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:14:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:14:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:14:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:14:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:14:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:14:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:14:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:14:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:14:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:14:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:14:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:14:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:14:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:14:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:14:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:14:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:14:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:14:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:14:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:14:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:14:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:14:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:15:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:15:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:15:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:15:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:15:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:15:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:15:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:15:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:15:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:15:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:15:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:15:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:15:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:15:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:15:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:15:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:15:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:15:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:15:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:15:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:15:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:15:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:15:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:15:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:15:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:15:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:15:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:15:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:15:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:15:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:15:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:15:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:15:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:15:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:15:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:15:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:15:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:15:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:15:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:15:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:15:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:15:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:15:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:15:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:15:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:15:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:15:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:15:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:15:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:15:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:15:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:15:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:15:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:15:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:15:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:15:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:15:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:15:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:15:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:15:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:15:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:15:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:15:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:15:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:15:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:15:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:15:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:15:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:15:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:15:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:15:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:15:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:15:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:15:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:16:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:16:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:16:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:16:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:16:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:16:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:16:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:16:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:16:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:16:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:16:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:16:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:16:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:16:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:16:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:16:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:16:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:16:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:16:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:16:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:16:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:16:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:16:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:16:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:16:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:16:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:16:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:16:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:16:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:16:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:16:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:16:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:16:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:16:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:16:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:16:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:16:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:16:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:16:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:16:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:16:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:16:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:16:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:16:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:16:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:16:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:16:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:16:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:16:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:16:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:16:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:16:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:16:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:16:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:16:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:16:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:16:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:16:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:16:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:16:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:16:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:16:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:16:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:16:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:16:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:16:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:16:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:16:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:16:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:16:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:16:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:16:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:16:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:17:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:17:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:17:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:17:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:17:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:17:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:17:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:17:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:17:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:17:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:17:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:17:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:17:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:17:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:17:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:17:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:17:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:17:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:17:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:17:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:17:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:17:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:17:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:17:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:17:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:17:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:17:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:17:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:17:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:17:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:17:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:17:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:17:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:17:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:17:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:17:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:17:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:17:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:17:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:17:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:17:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:17:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:17:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:17:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:17:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:17:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:17:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:17:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:17:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:17:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:17:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:17:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:17:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:17:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:17:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:17:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:17:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:17:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:17:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:17:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:17:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:17:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:17:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:18:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:18:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:18:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:18:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:18:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:18:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:18:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:18:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:18:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:18:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:18:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:18:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:18:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:18:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:18:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:18:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:18:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:18:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:18:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:18:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:18:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:18:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:18:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:18:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:18:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:18:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:18:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:18:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:18:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:18:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:18:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:18:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:18:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:18:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:18:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:18:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:18:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:18:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:18:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:18:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:18:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:18:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:18:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:18:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:18:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:18:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:18:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:18:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:18:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:18:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:18:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:18:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:18:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:18:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:18:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:18:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:18:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:18:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:18:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:18:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:18:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:18:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:18:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:18:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:18:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:18:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:18:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:18:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:18:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:18:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:18:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:18:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:18:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:19:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:19:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:19:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:19:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:19:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:19:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:19:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:19:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:19:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:19:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:19:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:19:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:19:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:19:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:19:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:19:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:19:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:19:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:19:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:19:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:19:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:19:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:19:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:19:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:19:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:19:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:19:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:19:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:19:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:19:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:19:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:19:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:19:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:19:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:19:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:19:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:19:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:19:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:19:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:19:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:19:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:19:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:19:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:19:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:19:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:19:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:19:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:19:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:19:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:19:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:19:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:19:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:19:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:19:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:19:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:19:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:19:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:19:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:19:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:19:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:19:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:19:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:19:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:19:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:19:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:19:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:19:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:19:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:19:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:20:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:20:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:20:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:20:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:20:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:20:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:20:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:20:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:20:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:20:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:20:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:20:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:20:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:20:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:20:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:20:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:20:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:20:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:20:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:20:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:20:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:20:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:20:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:20:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:20:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:20:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:20:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:20:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:20:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:20:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:20:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:20:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:20:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:20:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:20:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:20:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:20:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:20:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:20:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:20:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:20:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:20:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:20:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:20:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:20:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:20:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:20:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:20:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:20:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:20:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:20:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:20:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:20:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:20:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:20:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:20:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:20:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:20:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:20:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:20:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:20:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:20:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:20:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:20:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:20:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:20:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:20:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:20:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:20:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:20:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:20:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:20:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:20:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:20:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:21:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:21:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:21:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:21:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:21:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:21:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:21:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:21:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:21:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:21:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:21:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:21:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:21:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:21:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:21:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:21:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:21:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:21:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:21:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:21:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:21:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:21:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:21:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:21:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:21:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:21:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:21:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:21:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:21:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:21:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:21:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:21:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:21:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:21:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:21:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:21:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:21:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:21:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:21:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:21:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:21:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:21:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:21:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:21:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:21:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:21:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:21:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:21:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:21:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:21:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:21:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:21:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:21:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:21:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:21:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:21:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:21:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:21:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:21:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:21:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:21:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:21:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:21:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:21:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:21:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:21:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:21:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:21:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:21:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:21:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:21:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:21:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:21:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:22:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:22:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:22:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:22:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:22:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:22:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:22:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:22:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:22:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:22:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:22:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:22:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:22:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:22:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:22:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:22:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:22:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:22:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:22:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:22:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:22:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:22:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:22:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:22:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:22:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:22:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:22:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:22:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:22:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:22:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:22:22 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:22 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:22 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:22:22 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:22:22 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:22 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:32 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:22:32 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:22:32 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:32 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:22:32 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:22:32 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:32 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:22:32 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:22:32 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:22:32 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:22:32 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:32 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:22:32 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:32 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:32 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:32 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:22:32 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:22:32 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:42 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:22:42 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:22:42 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:22:42 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:42 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:42 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:22:42 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:22:42 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:22:42 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:22:42 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:22:42 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:22:42 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:42 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:42 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:22:42 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:42 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:42 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:22:42 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:52 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:22:52 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:22:52 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:22:52 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:22:52 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:22:52 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:52 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:22:52 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:52 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:22:52 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:22:52 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:22:52 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:22:52 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:22:52 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:52 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:22:52 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:22:52 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:22:52 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:02 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:23:02 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:23:02 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:02 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:23:02 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:23:02 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:23:02 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:23:02 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:23:02 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:23:02 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:02 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:23:02 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:02 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:23:02 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:23:02 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:02 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:23:02 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:02 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:12 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:23:12 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:23:12 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:23:12 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:23:12 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:12 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:23:12 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:23:12 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:12 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:23:12 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:23:12 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:23:12 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:12 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:23:12 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:12 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:23:12 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:12 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:23:12 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:22 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:23:22 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:23:22 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:23:22 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:22 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:23:22 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:23:22 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:23:22 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:23:22 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:23:22 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:22 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:23:22 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:23:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:23:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:23:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:23:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:23:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:23:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:23:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:23:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:23:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:23:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:23:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:23:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:23:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:23:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:23:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:23:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:23:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:23:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:23:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:23:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:23:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:23:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:23:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:23:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:23:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:23:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:23:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:23:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:23:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:23:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:23:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:23:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:23:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:23:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:23:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:23:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:23:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:23:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:23:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:23:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:24:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:24:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:24:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:24:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:24:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:24:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:24:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:24:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:24:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:24:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:24:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:24:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:24:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:24:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:24:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:24:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:24:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:24:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:24:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:24:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:13 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:13 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:13 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:24:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:24:13 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:23 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:24:23 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:24:23 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:24:23 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:23 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:24:23 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:24:23 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:23 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:24:23 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:24:23 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:24:23 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:24:23 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:23 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:23 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:23 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:24:23 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:24:23 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:23 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:33 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:24:33 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:33 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:24:33 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:33 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:24:33 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:24:33 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:24:33 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:24:33 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:24:33 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:33 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:24:33 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:24:33 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:33 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:33 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:33 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:24:33 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:24:33 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:43 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:24:43 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:24:43 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:43 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:24:43 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:43 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:24:43 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:24:43 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:43 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:24:43 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:24:43 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:24:43 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:24:43 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:43 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:43 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:24:43 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:43 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:43 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:24:53 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:24:53 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:24:53 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:24:53 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:24:53 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:24:53 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:24:53 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:24:53 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:24:53 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:24:53 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:53 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:53 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:24:53 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:24:53 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:24:53 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:53 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:24:53 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:24:53 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:03 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:25:03 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:25:03 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:03 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:25:03 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:25:03 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:03 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:25:03 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:25:03 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:25:03 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:25:03 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:25:03 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:03 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:03 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:25:03 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:03 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:25:03 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:03 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:25:13 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:25:13 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:25:13 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:13 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:25:13 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:25:13 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:25:13 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:25:13 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:13 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:25:13 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:25:13 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:25:13 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:13 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:13 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:25:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:25:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:25:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:25:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:25:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:25:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:25:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:25:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:25:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:25:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:25:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:25:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:25:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:25:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:25:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:25:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:25:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:25:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:25:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:25:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:25:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:25:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:25:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:25:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:25:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:25:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:25:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:25:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:25:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:25:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:25:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:25:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:25:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:25:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:25:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:25:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:25:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:25:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:25:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:25:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:25:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:25:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:25:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:25:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:25:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:25:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:25:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:25:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:25:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:25:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:25:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:25:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:25:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:26:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:26:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:26:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:26:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:26:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:26:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:26:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:26:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:26:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:26:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:26:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:04 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:26:14 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:26:14 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:26:14 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:26:14 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:26:14 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:14 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:26:14 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:26:14 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:26:14 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:14 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:14 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:26:14 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:26:14 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:14 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:26:14 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:14 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:26:14 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:26:14 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:24 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:26:24 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:26:24 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:24 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:24 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:26:24 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:26:24 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:26:24 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:24 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:26:24 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:26:24 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:26:24 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:26:24 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:26:24 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:24 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:24 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:26:24 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:26:24 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:34 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:26:34 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:34 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:26:34 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:26:34 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:26:34 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:26:34 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:34 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:34 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:26:34 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:26:34 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:26:34 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:26:34 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:34 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:34 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:26:34 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:26:34 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:34 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:26:44 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:26:44 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:26:44 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:26:44 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:26:44 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:26:44 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:26:44 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:44 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:44 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:26:44 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:26:44 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:44 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:26:44 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:26:44 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:26:44 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:44 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:26:44 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:44 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:54 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:26:54 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:54 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:26:54 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:54 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:26:54 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:26:54 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:26:54 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:26:54 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:26:54 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:26:54 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:26:54 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:26:54 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:26:54 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:26:54 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:54 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:26:54 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:26:54 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:04 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:27:04 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:27:04 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:27:04 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:27:04 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:04 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:27:04 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:27:04 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:04 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:27:04 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:27:04 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:27:04 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:04 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:27:04 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:04 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:04 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:27:04 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:27:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:27:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:27:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:27:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:27:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:27:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:27:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:27:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:27:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:27:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:27:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:27:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:27:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:27:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:27:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:27:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:27:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:27:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:27:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:27:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:27:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:27:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:27:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:27:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:27:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:27:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:27:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:27:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:27:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:27:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:27:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:27:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:27:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:27:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:27:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:27:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:27:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:27:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:27:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:27:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:27:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:27:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:27:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:27:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:27:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:27:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:27:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:27:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:27:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:27:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:27:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:27:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:27:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:27:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:27:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:27:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:27:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:27:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:27:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:27:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:27:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:27:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:27:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:28:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:28:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:28:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:28:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:28:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:28:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:28:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:28:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:28:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:05 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:28:05 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:05 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:05 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:28:05 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:05 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:28:15 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:28:15 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:28:15 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:28:15 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:15 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:28:15 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:28:15 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:28:15 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:15 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:28:15 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:15 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:28:15 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:28:15 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:28:15 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:15 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:28:15 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:15 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:15 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:28:25 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:28:25 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:28:25 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:28:25 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:28:25 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:28:25 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:28:25 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:25 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:28:25 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:28:25 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:25 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:28:25 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:25 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:25 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:28:25 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:25 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:25 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:28:25 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:28:35 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:28:35 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:28:35 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:28:35 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:28:35 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:28:35 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:35 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:35 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:28:35 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:28:35 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:28:35 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:35 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:28:35 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:28:35 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:28:35 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:35 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:28:35 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:35 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:45 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:28:45 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:28:45 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:28:45 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:28:45 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:28:45 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:28:45 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:28:45 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:45 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:28:45 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:45 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:28:45 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:45 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:45 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:45 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:45 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:28:45 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:28:45 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:28:55 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:28:55 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:28:55 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:28:55 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:28:55 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:28:55 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:55 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:28:55 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:28:55 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:28:55 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:55 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:28:55 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:28:55 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:28:55 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:55 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:55 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:28:55 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:28:55 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:05 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:29:05 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:29:05 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:29:05 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:29:05 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:29:05 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:05 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:29:05 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:29:05 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:29:05 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:29:05 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:05 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:29:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:29:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:29:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:29:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:29:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:29:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:29:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:29:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:29:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:29:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:29:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:29:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:29:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:29:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:29:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:29:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:29:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:29:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:29:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:29:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:29:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:29:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:29:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:29:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:29:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:29:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:29:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:29:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:29:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:29:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:29:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:29:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:29:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:29:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:29:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:29:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:29:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:29:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:29:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:29:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:29:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:29:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:29:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:29:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:29:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:29:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:29:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:29:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:29:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:29:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:29:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:29:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:29:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:29:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:29:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:29:56 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:29:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:29:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:56 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:29:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:29:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:06 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:30:06 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:30:06 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:06 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:30:06 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:30:06 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:06 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:30:06 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:30:06 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:30:06 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:30:06 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:06 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:30:06 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:30:06 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:06 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:06 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:30:06 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:06 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:30:16 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:30:16 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:16 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:30:16 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:30:16 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:30:16 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:30:16 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:16 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:30:16 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:30:16 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:16 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:30:16 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:30:16 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:16 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:30:16 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:16 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:30:16 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:16 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:30:26 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:30:26 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:30:26 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:26 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:30:26 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:30:26 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:30:26 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:30:26 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:30:26 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:30:26 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:26 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:26 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:30:26 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:30:26 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:30:26 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:30:26 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:26 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:26 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:36 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:30:36 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:30:36 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:30:36 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:30:36 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:30:36 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:30:36 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:30:36 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:36 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:36 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:30:36 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:30:36 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:36 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:36 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:36 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:30:36 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:30:36 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:30:36 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:46 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:30:46 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:30:46 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:30:46 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:46 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:30:46 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:30:46 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:30:46 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:30:46 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:46 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:46 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:30:46 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:30:46 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:30:46 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:46 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:46 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:30:46 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:46 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:30:56 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:30:56 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:30:56 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:30:56 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:56 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:30:56 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:30:56 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:30:56 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:56 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:30:56 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:30:56 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:30:56 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:30:56 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:56 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:56 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:30:56 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:30:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:30:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:31:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:31:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:31:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:31:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:31:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:31:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:31:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:31:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:31:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:31:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:31:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:31:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:31:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:31:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:31:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:31:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:31:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:31:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:31:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:31:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:31:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:31:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:31:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:31:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:31:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:31:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:31:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:31:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:31:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:31:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:31:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:31:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:31:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:31:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:31:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:31:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:31:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:31:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:31:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:31:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:31:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:31:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:31:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:31:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:31:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:31:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:31:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:31:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:31:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:31:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:31:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:31:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:31:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:47 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:47 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:47 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:31:47 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:31:47 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:47 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:57 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:31:57 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:31:57 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:31:57 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:31:57 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:31:57 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:57 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:31:57 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:57 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:31:57 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:31:57 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:31:57 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:31:57 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:31:57 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:57 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:31:57 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:31:57 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:31:57 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:07 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:32:07 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:32:07 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:32:07 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:32:07 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:32:07 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:07 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:32:07 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:07 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:32:07 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:32:07 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:32:07 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:07 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:32:07 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:07 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:32:07 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:07 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:07 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:32:17 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:32:17 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:32:17 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:32:17 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:32:17 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:17 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:32:17 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:32:17 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:17 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:32:17 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:17 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:32:17 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:32:17 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:32:17 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:17 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:17 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:17 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:32:17 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:32:27 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:32:27 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:32:27 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:32:27 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:32:27 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:32:27 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:32:27 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:27 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:27 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:32:27 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:32:27 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:27 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:32:27 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:27 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:32:27 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:32:27 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:32:27 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:27 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:37 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:32:37 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:32:37 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:37 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:32:37 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:32:37 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:37 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:32:37 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:32:37 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:32:37 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:32:37 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:37 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:32:37 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:37 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:37 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:32:37 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:32:37 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:32:37 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:47 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:32:47 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:32:47 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:32:47 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:32:47 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:32:47 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:47 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:47 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:32:47 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:32:47 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:32:47 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:32:47 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:32:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:32:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:32:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:32:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:32:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:32:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:32:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:32:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:32:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:32:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:32:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:32:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:32:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:32:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:32:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:32:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:32:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:33:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:33:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:33:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:33:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:33:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:33:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:33:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:33:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:33:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:33:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:33:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:33:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:33:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:33:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:33:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:33:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:33:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:33:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:33:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:33:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:33:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:33:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:33:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:33:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:33:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:33:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:33:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:33:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:33:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:33:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:33:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:33:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:33:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:28 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:28 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:28 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:33:28 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:33:28 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:33:38 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:33:38 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:33:38 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:33:38 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:33:38 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:38 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:33:38 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:38 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:33:38 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:33:38 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:33:38 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:33:38 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:38 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:33:38 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:38 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:33:38 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:38 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:33:38 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:48 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:33:48 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:33:48 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:33:48 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:48 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:33:48 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:33:48 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:33:48 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:48 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:33:48 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:33:48 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:33:48 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:48 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:48 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:48 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:33:48 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:33:48 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:48 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:33:58 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:33:58 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:33:58 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:58 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:33:58 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:58 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:33:58 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:33:58 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:33:58 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:33:58 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:33:58 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:33:58 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:33:58 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:58 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:58 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:33:58 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:33:58 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:33:58 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:34:08 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:34:08 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:34:08 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:34:08 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:34:08 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:34:08 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:34:08 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:34:08 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:34:08 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:08 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:34:08 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:08 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:08 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:08 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:34:08 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:34:08 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:08 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:34:08 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:18 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:34:18 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:34:18 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:34:18 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:18 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:18 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:34:18 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:34:18 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:18 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:34:18 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:34:18 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:34:18 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:34:18 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:18 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:18 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:34:18 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:18 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:34:18 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:34:28 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:34:28 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:34:28 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:34:28 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:28 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:34:28 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:34:28 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:34:28 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:34:28 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:28 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:28 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:34:28 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:34:28 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:34:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:34:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:34:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:34:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:34:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:34:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:34:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:34:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:34:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:34:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:34:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:34:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:34:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:34:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:34:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:34:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:34:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:34:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:34:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:34:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:34:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:34:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:34:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:34:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:34:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:34:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:34:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:34:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:34:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:34:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:34:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:34:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:34:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:34:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:34:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:34:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:34:59 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:34:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:34:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:34:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:09 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:35:09 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:35:09 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:09 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:35:09 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:35:09 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:35:09 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:35:09 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:09 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:35:09 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:35:09 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:35:09 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:09 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:09 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:35:09 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:09 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:09 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:35:09 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:19 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:35:19 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:35:19 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:35:19 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:35:19 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:19 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:19 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:35:19 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:19 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:35:19 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:35:19 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:35:19 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:35:19 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:19 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:35:19 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:19 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:19 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:19 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:35:29 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:35:29 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:35:29 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:29 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:35:29 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:35:29 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:35:29 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:35:29 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:29 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:35:29 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:29 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:35:29 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:35:29 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:29 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:29 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:35:29 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:29 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:29 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:35:39 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:35:39 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:35:39 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:35:39 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:35:39 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:39 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:35:39 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:39 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:35:39 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:35:39 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:35:39 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:35:39 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:39 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:35:39 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:39 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:39 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:39 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:39 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:35:49 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:35:49 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:35:49 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:35:49 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:35:49 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:35:49 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:49 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:35:49 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:49 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:35:49 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:35:49 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:49 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:35:49 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:49 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:49 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:49 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:35:49 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:49 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:35:59 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:35:59 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:35:59 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:59 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:35:59 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:35:59 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:35:59 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:59 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:35:59 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:35:59 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:35:59 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:35:59 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:35:59 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:59 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:35:59 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:35:59 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:35:59 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:36:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:36:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:36:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:36:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:36:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:36:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:36:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:36:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:36:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:36:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:36:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:36:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:36:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:36:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:36:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:36:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:36:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:36:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:36:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:36:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:36:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:36:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:36:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:36:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:36:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:36:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:36:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:36:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:36:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:36:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:36:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:36:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:36:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:36:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:36:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:36:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:36:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:36:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:36:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:36:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:36:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:36:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:36:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:36:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:36:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:36:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:36:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:36:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:36:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:36:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:36:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:36:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:36:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:36:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:36:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:36:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:36:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:36:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:36:50 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:50 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:50 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:36:50 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:36:50 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:36:50 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:37:00 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:37:00 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:37:00 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:37:00 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:37:00 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:00 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:37:00 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:37:00 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:37:00 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:00 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:37:00 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:37:00 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:00 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:00 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:00 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:37:00 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:37:00 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:37:00 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:10 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:37:10 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:37:10 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:37:10 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:10 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:37:10 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:10 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:37:10 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:37:10 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:37:10 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:10 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:37:10 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:37:10 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:37:10 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:37:10 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:10 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:10 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:10 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:37:20 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:37:20 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:37:20 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:37:20 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:20 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:37:20 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:20 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:37:20 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:37:20 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:37:20 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:37:20 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:20 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:37:20 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:37:20 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:20 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:20 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:37:20 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:37:20 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:30 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:37:30 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:37:30 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:37:30 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:30 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:37:30 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:37:30 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:37:30 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:30 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:37:30 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:37:30 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:30 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:37:30 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:37:30 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:30 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:37:30 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:30 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:30 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:37:40 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:37:40 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:37:40 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:40 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:37:40 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:37:40 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:40 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:37:40 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:37:40 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:37:40 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:40 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:37:40 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:37:40 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:37:40 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:37:40 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:40 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:37:40 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:40 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:37:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:37:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:37:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:37:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:37:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:37:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:37:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:37:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:37:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:37:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:37:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:37:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:37:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:38:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:38:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:38:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:38:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:38:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:38:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:38:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:38:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:38:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:01 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:01 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:01 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:38:01 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:01 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:38:01 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:11 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:38:11 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:38:11 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:38:11 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:11 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:11 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:38:11 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:38:11 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:38:11 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:38:11 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:38:11 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:38:11 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:11 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:11 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:11 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:11 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:11 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:38:11 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:38:21 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:38:21 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:38:21 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:21 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:21 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:38:21 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:38:21 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:38:21 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:38:21 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:38:21 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:38:21 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:38:21 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:21 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:21 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:38:21 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:38:21 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:21 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:21 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:31 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:38:31 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:38:31 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:31 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:38:31 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:38:31 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:38:31 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:38:31 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:31 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:38:31 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:38:31 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:38:31 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:31 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:31 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:31 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:31 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:38:31 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:38:31 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:41 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:38:41 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:38:41 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:38:41 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:38:41 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:41 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:41 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:41 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:38:41 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:38:41 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:38:41 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:38:41 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:38:41 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:38:41 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:41 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:38:41 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:41 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:41 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:51 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:38:51 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:38:51 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:51 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:51 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:38:51 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:38:51 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:38:51 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:38:51 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:38:51 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:38:51 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:38:51 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:38:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-16 06:38:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:38:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-16 06:38:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403" 2025-09-16 06:38:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-16 06:39:01 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-16 06:39:01 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-16 06:39:01 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:39:01 TRC Executing Prometheus query: DS Identity backend connections 2025-09-16 06:39:01 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-16 06:39:01 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-16 06:39:01 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:39:01 TRC Executing Prometheus query: container_memory_failcnt 2025-09-16 06:39:01 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-16 06:39:01 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-16 06:39:01 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-16 06:39:01 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-16 06:39:01 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 06:39:01 INF Re-authenticating Prometheus client 2025-09-16 06:39:01 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 06:39:01 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 06:39:01 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 06:39:01 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 06:39:01 WRN Prometheus returned 401. Re-authenticating... 2025-09-16 12:40:02 INF Adding event: Scenario_benchmark-stop 2025-09-16 13:05:47 INF Stopping Lodemon 2025-09-16 13:05:47 INF Stopping monitoring service...