==================================================================================================== ============================ Pod describe lodemon-6fddcf76bb-nj6jh ============================ ==================================================================================================== Name: lodemon-6fddcf76bb-nj6jh Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-9d1e5395-5mf3/10.142.0.104 Start Time: Thu, 11 Sep 2025 00:40:05 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=6fddcf76bb skaffold.dev/run-id=29b21090-a82e-40d5-ada1-801ceb915cf0 Annotations: Status: Running IP: 10.106.40.152 IPs: IP: 10.106.40.152 Controlled By: ReplicaSet/lodemon-6fddcf76bb Containers: lodemon: Container ID: containerd://c9390f299de1ba35dc18479590af055757eeb0a55429265c5e6592914b1b20e5 Image: gcr.io/engineeringpit/performance-dashboard/lodemon:latest Image ID: gcr.io/engineeringpit/performance-dashboard/lodemon@sha256:0cb8d145a112e633f26bc9a523cf5650a303007a3767cb1e83c5fc180881d0f1 Port: 8080/TCP Host Port: 0/TCP State: Running Started: Thu, 11 Sep 2025 00:40:25 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: MONITORING_CONFIG_PATH: /cfg/lodemon_config.yaml Mounts: /cfg/ from lodemon-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qrxzd (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: lodemon-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false kube-api-access-qrxzd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 45s default-scheduler Successfully assigned xlou/lodemon-6fddcf76bb-nj6jh to gke-xlou-cdm-default-pool-9d1e5395-5mf3 Normal Pulling 46s kubelet Pulling image "gcr.io/engineeringpit/performance-dashboard/lodemon:latest" Normal Pulled 26s kubelet Successfully pulled image "gcr.io/engineeringpit/performance-dashboard/lodemon:latest" in 19.883s (19.883s including waiting). Image size: 689741197 bytes. Normal Created 26s kubelet Created container: lodemon Normal Started 26s kubelet Started container lodemon ==================================================================================================== ============================== Pod logs lodemon-6fddcf76bb-nj6jh ============================== ==================================================================================================== 2025-09-11 00:40:25 INF Starting lodemon server on port 8080 2025-09-11 00:40:25 INF Initializing clients... 2025-09-11 00:40:25 DBG Fetching ID token for accessing prometheus 2025-09-11 00:40:25 DBG Fetched ID token successfully 2025-09-11 00:40:25 INF Clients initialized successfully 2025-09-11 00:40:50 INF Configuring Lodemon 2025-09-11 00:40:50 INF Configuring monitoring units 2025-09-11 00:40:50 INF Prometheus config: {https://api.pre-prod-monitor.forgeblocks.com/ 172791522915-3e3324sfe2hp8olcsc700a34ckn6pjhu.apps.googleusercontent.com [{DS Java heap mem usage ds_jvm_memory_heap_used_bytes [] } {IDM Java heap mem usage idm_jvm_memory_usage_heap_usage [] } {AM Java heap mem usage am_jvm_memory_usage_heap_usage [] }] [{Kubelet Volume Stats Inodes kubelet_volume_stats_inodes [] } { container_memory_failcnt [] }] [{DS Identity backend connections ds_backend_db_cache_size_total_bytes [{backend amIdentityStore}] fr-platform}]} 2025-09-11 00:40:50 TRC Parsing Prometheus queries from config 2025-09-11 00:40:50 TRC Parsing component query: DS Java heap mem usage 2025-09-11 00:40:50 TRC Parsing component query: IDM Java heap mem usage 2025-09-11 00:40:50 TRC Parsing component query: AM Java heap mem usage 2025-09-11 00:40:50 TRC Parsing global query: Kubelet Volume Stats Inodes 2025-09-11 00:40:50 TRC Parsing global query: 2025-09-11 00:40:50 TRC Parsing scoped query: DS Identity backend connections 2025-09-11 00:40:50 INF Prometheus monitoring unit is enabled. Configuring prometheus units 2025-09-11 00:40:50 DBG - Configuring prometheus unit: DS Java heap mem usage 2025-09-11 00:40:50 DBG - Prometheus unit configured: DS Java heap mem usage 2025-09-11 00:40:50 DBG - Configuring prometheus unit: IDM Java heap mem usage 2025-09-11 00:40:50 DBG - Prometheus unit configured: IDM Java heap mem usage 2025-09-11 00:40:50 DBG - Configuring prometheus unit: AM Java heap mem usage 2025-09-11 00:40:50 DBG - Prometheus unit configured: AM Java heap mem usage 2025-09-11 00:40:50 DBG - Configuring prometheus unit: Kubelet Volume Stats Inodes 2025-09-11 00:40:50 DBG - Prometheus unit configured: Kubelet Volume Stats Inodes 2025-09-11 00:40:50 DBG - Configuring prometheus unit: container_memory_failcnt 2025-09-11 00:40:50 DBG - Prometheus unit configured: container_memory_failcnt 2025-09-11 00:40:50 DBG - Configuring prometheus unit: DS Identity backend connections 2025-09-11 00:40:50 DBG - Prometheus unit configured: DS Identity backend connections 2025-09-11 00:40:50 INF  - All monitoring units configured 2025-09-11 00:40:50 INF Configuring monitoring service 2025-09-11 00:40:50 INF  - Adding monitoring units to the service 2025-09-11 00:40:50 DBG - Adding monitoring unit: DS Java heap mem usage 2025-09-11 00:40:50 DBG - Adding monitoring unit: IDM Java heap mem usage 2025-09-11 00:40:50 DBG - Adding monitoring unit: AM Java heap mem usage 2025-09-11 00:40:50 DBG - Adding monitoring unit: Kubelet Volume Stats Inodes 2025-09-11 00:40:50 DBG - Adding monitoring unit: container_memory_failcnt 2025-09-11 00:40:50 DBG - Adding monitoring unit: DS Identity backend connections 2025-09-11 00:40:50 INF  - All monitoring units added to the service 2025-09-11 00:40:50 INF Starting Lodemon now 2025-09-11 00:40:50 INF Starting monitoring service... 2025-09-11 00:40:50 TRC Executing Prometheus query: DS Identity backend connections 2025-09-11 00:40:50 TRC Running prometheus query: ds_backend_db_cache_size_total_bytes{namespace="fr-platform",backend="amIdentityStore",cluster="perf-haproxy"} 2025-09-11 00:40:50 TRC Executing Prometheus query: AM Java heap mem usage 2025-09-11 00:40:50 TRC Executing Prometheus query: DS Java heap mem usage 2025-09-11 00:40:50 TRC Running prometheus query: am_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-11 00:40:50 TRC Running prometheus query: ds_jvm_memory_heap_used_bytes{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-11 00:40:50 TRC Executing Prometheus query: Kubelet Volume Stats Inodes 2025-09-11 00:40:50 TRC Running prometheus query: kubelet_volume_stats_inodes{cluster="perf-haproxy"} 2025-09-11 00:40:50 TRC Executing Prometheus query: IDM Java heap mem usage 2025-09-11 00:40:50 TRC Executing Prometheus query: container_memory_failcnt 2025-09-11 00:40:50 TRC Running prometheus query: idm_jvm_memory_usage_heap_usage{namespace="fr-platform",cluster="perf-haproxy"} 2025-09-11 00:40:50 TRC Running prometheus query: container_memory_failcnt{cluster="perf-haproxy"} 2025-09-11 00:40:51 ERR Error executing Prometheus query: DS Identity backend connections error="Query error: client_error: client error: 403" 2025-09-11 00:40:51 ERR Error executing Prometheus query: DS Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-11 00:40:51 ERR Error executing Prometheus query: AM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-11 00:40:51 ERR Error executing Prometheus query: IDM Java heap mem usage error="Query error: client_error: client error: 403" 2025-09-11 00:40:51 ERR Error executing Prometheus query: Kubelet Volume Stats Inodes error="Query error: client_error: client error: 403" 2025-09-11 00:40:51 ERR Error executing Prometheus query: container_memory_failcnt error="Query error: client_error: client error: 403"