==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: am-66bfc88d64-q2t85 Namespace: xlou Priority: 0 Node: gke-xlou-cdm-default-pool-33cc2627-pkz3/10.142.15.196 Start Time: Tue, 20 Sep 2022 01:54:50 +0000 Labels: app=am app.kubernetes.io/component=am app.kubernetes.io/instance=am app.kubernetes.io/name=am app.kubernetes.io/part-of=forgerock pod-template-hash=66bfc88d64 skaffold.dev/profile=medium skaffold.dev/run-id=xlou tier=middle Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true Status: Running IP: 10.0.3.10 IPs: IP: 10.0.3.10 Controlled By: ReplicaSet/am-66bfc88d64 Init Containers: fbc-init: Container ID: containerd://3f378a13de3a6c9b6725df43bd53debc992c3c218ac5ae3fce164f2f5ade0f5c Image: gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7 Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7 Port: Host Port: Command: /bin/bash -c if [ -d /fbc/config ]; then echo "Existing openam configuration found. Skipping copy" else echo "Copying docker image configuration files to the shared volume" cd /home/forgerock/openam cp -r .homeVersion * /fbc fi State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 20 Sep 2022 01:54:51 +0000 Finished: Tue, 20 Sep 2022 01:54:51 +0000 Ready: True Restart Count: 0 Environment: Mounts: /fbc from fbc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-42mlv (ro) truststore-init: Container ID: containerd://9af4a6e72de6619dae14e542d6e1621de838f6811bbf16a84fbabaf44a5e8380 Image: gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7 Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7 Port: Host Port: Command: /home/forgerock/import-pem-certs.sh State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 20 Sep 2022 01:54:52 +0000 Finished: Tue, 20 Sep 2022 01:54:52 +0000 Ready: True Restart Count: 0 Environment: TRUSTSTORE_PATH: /truststore/amtruststore TRUSTSTORE_PASSWORD: changeit AM_PEM_TRUSTSTORE: /var/run/secrets/truststore/cacerts Mounts: /truststore from new-truststore (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-42mlv (ro) /var/run/secrets/truststore from truststore (rw) Containers: openam: Container ID: containerd://698b3831eefc4766a2cac6679afb2e7b006bd6028d91c70a3f8831af4a32127d Image: gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7 Image ID: gcr.io/engineeringpit/lodestar-images/am@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7 Port: 8080/TCP Host Port: 0/TCP Command: bash /home/forgerock/docker-entrypoint.sh State: Running Started: Tue, 20 Sep 2022 01:54:53 +0000 Ready: True Restart Count: 0 Limits: memory: 10Gi Requests: cpu: 11 memory: 10Gi Liveness: http-get http://:8080/am/json/health/live delay=30s timeout=5s period=30s #success=1 #failure=3 Readiness: http-get http://:8080/am/json/health/ready delay=20s timeout=5s period=10s #success=1 #failure=3 Startup: http-get http://:8080/am/json/health/live delay=0s timeout=1s period=10s #success=1 #failure=40 Environment Variables from: am-env-secrets Secret Optional: false platform-config ConfigMap Optional: false amster-env-secrets Secret Optional: false ds-env-secrets Secret Optional: false Environment: NAMESPACE: xlou (v1:metadata.namespace) AM_STORES_SSL_ENABLED: true TRUSTSTORE_PATH: /home/forgerock/amtruststore TRUSTSTORE_PASSWORD: changeit AM_STORES_USER_TYPE: LDAPv3ForForgeRockIAM Mounts: /home/forgerock/amtruststore from new-truststore (ro,path="amtruststore") /home/forgerock/openam from fbc (rw) /var/run/secrets/am from am-secrets (rw) /var/run/secrets/amster from amster-key (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-42mlv (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: fbc: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: am-secrets: Type: Projected (a volume that contains injected data from multiple sources) SecretName: am-passwords SecretOptionalName: SecretName: am-keystore SecretOptionalName: amster-key: Type: Secret (a volume populated by a Secret) SecretName: amster Optional: false truststore: Type: Secret (a volume populated by a Secret) SecretName: truststore-pem Optional: false new-truststore: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-42mlv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m20s default-scheduler Successfully assigned xlou/am-66bfc88d64-q2t85 to gke-xlou-cdm-default-pool-33cc2627-pkz3 Normal Pulling 3m20s kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7" Normal Pulled 3m19s kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7" in 239.581176ms Normal Created 3m19s kubelet Created container fbc-init Normal Started 3m19s kubelet Started container fbc-init Normal Pulling 3m19s kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7" Normal Pulling 3m18s kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7" Normal Pulled 3m18s kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7" in 238.386278ms Normal Started 3m18s kubelet Started container truststore-init Normal Created 3m18s kubelet Created container truststore-init Normal Pulled 3m17s kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/am:xlou@sha256:5d7ce8a70fd2e12167c404ce4da3bf4d49721d2e9c3847436343ddbfb7d4b4e7" in 213.959756ms Normal Created 3m17s kubelet Created container openam Normal Started 3m17s kubelet Started container openam Warning Unhealthy 2m20s (x6 over 3m10s) kubelet Startup probe failed: Get "http://10.0.3.10:8080/am/json/health/live": dial tcp 10.0.3.10:8080: connect: connection refused Warning Unhealthy 119s (x2 over 2m9s) kubelet Startup probe failed: Get "http://10.0.3.10:8080/am/json/health/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers) ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== Copying docker image configuration files to the shared volume Copying /usr/local/openjdk-11/lib/security/cacerts to /truststore/amtruststore Found (1) certificates in /var/run/secrets/truststore/cacerts Importing (1) certificates into /truststore/amtruststore Certificate was added to keystore Import complete! Copying bootstrap files for legacy AMKeyProvider Waiting for User Store to be available Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Trying ds-idrepo-1.ds-idrepo:8080/alive endpoint Trying ds-idrepo-2.ds-idrepo:8080/alive endpoint Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint User Store is responding Waiting for CTS to be available Trying ds-cts-0.ds-cts:8080/alive endpoint CTS is responding Waiting for Application Store to be available Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Application Store is responding Waiting for Policy Store to be available Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint Policy Store is responding Waiting for UMA Store to be available Trying ds-idrepo-0.ds-idrepo:8080/alive endpoint UMA Store is responding Starting tomcat with opts: -server -Dcom.sun.services.debug.mergeall=on -Dcom.sun.identity.configuration.directory=/home/forgerock/openam -Dcom.iplanet.services.stats.state=off -Dcom.sun.identity.sm.sms_object_filebased_enabled=true -Dorg.forgerock.donotupgrade=true -DtomcatAccessLogDir=/proc/self/fd -DtomcatAccessLogFile=1 -XX:MaxRAMPercentage=75 -XX:InitialRAMPercentage=75 -XX:MaxTenuringThreshold=1 -Djava.security.egd=file:/dev/urandom -Dorg.forgerock.am.oauth2.consent.xui_path=/OAuth2_XUI -Dcom.forgerock.am.fileconfig.basepaths=/home/forgerock/base/config/services,/home/forgerock/cdk/config/services -Djavax.net.ssl.trustStore=/home/forgerock/amtruststore -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=jks NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED 20-Sep-2022 01:55:51.037 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.65 20-Sep-2022 01:55:51.039 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jul 14 2022 12:28:53 UTC 20-Sep-2022 01:55:51.039 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.65.0 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 5.4.188+ 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/local/openjdk-11 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.16+8 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/local/tomcat 20-Sep-2022 01:55:51.040 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/local/tomcat 20-Sep-2022 01:55:51.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED 20-Sep-2022 01:55:51.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED 20-Sep-2022 01:55:51.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.services.debug.mergeall=on 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.identity.configuration.directory=/home/forgerock/openam 20-Sep-2022 01:55:51.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.iplanet.services.stats.state=off 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.identity.sm.sms_object_filebased_enabled=true 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.forgerock.donotupgrade=true 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -DtomcatAccessLogDir=/proc/self/fd 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -DtomcatAccessLogFile=1 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxRAMPercentage=75 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:InitialRAMPercentage=75 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:MaxTenuringThreshold=1 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.egd=file:/dev/urandom 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.forgerock.am.oauth2.consent.xui_path=/OAuth2_XUI 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.forgerock.am.fileconfig.basepaths=/home/forgerock/base/config/services,/home/forgerock/cdk/config/services 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStore=/home/forgerock/amtruststore 20-Sep-2022 01:55:51.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStorePassword=changeit 20-Sep-2022 01:55:51.051 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djavax.net.ssl.trustStoreType=jks 20-Sep-2022 01:55:51.051 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs= 20-Sep-2022 01:55:51.051 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat 20-Sep-2022 01:55:51.051 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat 20-Sep-2022 01:55:51.051 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp 20-Sep-2022 01:55:51.054 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.35] using APR version [1.7.0]. 20-Sep-2022 01:55:51.054 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true]. 20-Sep-2022 01:55:51.054 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 20-Sep-2022 01:55:51.057 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1n 15 Mar 2022] 20-Sep-2022 01:55:51.284 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"] 20-Sep-2022 01:55:51.301 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8081"] 20-Sep-2022 01:55:51.302 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [417] milliseconds 20-Sep-2022 01:55:51.343 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina] 20-Sep-2022 01:55:51.343 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.65] 20-Sep-2022 01:55:51.348 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/am] 20-Sep-2022 01:55:59.082 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Starting up OpenAM at Sep 20, 2022, 1:56:01 AM {"timestamp":"2022-09-20T01:56:04.552Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ldapClient=LdapClient(host=ds-idrepo-2.ds-idrepo, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Host 'ds-idrepo-2.ds-idrepo' cannot be resolved","context":"default"} {"timestamp":"2022-09-20T01:56:04.823Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ldapClient=LdapClient(host=ds-idrepo-2.ds-idrepo, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Host 'ds-idrepo-2.ds-idrepo' cannot be resolved","context":"default"} {"timestamp":"2022-09-20T01:56:04.895Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"org.forgerock.openam.plugins.PluginLifecycle","message":"PluginTools#upgradePlugins: Encountered plugin org.forgerock.openam.auth.nodes.CloudOnlyNodesPlugin not configured in SMS","context":"default"} {"timestamp":"2022-09-20T01:56:07.141Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringConfig","message":"JDMK runtime not found - Legacy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:56:07.155Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:56:07.218Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:56:09.240Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ldapClient=LdapClient(host=UNUSED.IN.FBC, port=9999, protocol=LDAP (clear-text)))' is no longer operational: Connect Error: Host 'UNUSED.IN.FBC' cannot be resolved","context":"default"} {"timestamp":"2022-09-20T01:56:09.303Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.sm.ldap.LDAPEventManager","message":"LDAPEventManager: received all entries changed event from EventService","context":"default"} {"timestamp":"2022-09-20T01:56:09.354Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.sm.ldap.SMSLdapObject","message":"SMSLDAPObject: got notifications, all objects changed","context":"default"} {"timestamp":"2022-09-20T01:56:10.070Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.sm.ldap.LDAPEventManager","message":"LDAPEventManager: received all entries changed event from EventService","context":"default"} {"timestamp":"2022-09-20T01:56:10.088Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.sm.ldap.SMSLdapObject","message":"SMSLDAPObject: got notifications, all objects changed","context":"default"} {"timestamp":"2022-09-20T01:56:10.301Z","level":"WARN","thread":"RADIUS-RadiusServerManager","logger":"org.forgerock.openam.radius.server.config.RadiusServerManager","message":"RADIUS service is not enabled.","context":"default"} {"timestamp":"2022-09-20T01:56:10.605Z","level":"WARN","thread":"Thread-5","logger":"org.forgerock.openam.services.push.PushNotificationService","message":"No Push Notification Service Config found for realm /","context":"default","exception":"java.util.NoSuchElementException: No value present\n\tat java.base/java.util.Optional.get(Optional.java:148)\n\tat org.forgerock.openam.services.push.PushNotificationServiceConfigHelperFactory.getConfigHelperFor(PushNotificationServiceConfigHelperFactory.java:58)\n\tat org.forgerock.openam.services.push.PushNotificationService.getConfigHelper(PushNotificationService.java:280)\n\tat org.forgerock.openam.services.push.PushNotificationService.updatePreferences(PushNotificationService.java:242)\n\tat org.forgerock.openam.services.push.PushNotificationService.init(PushNotificationService.java:157)\n\tat org.forgerock.openam.services.push.PushNotificationServiceSetupListener$1.run(PushNotificationServiceSetupListener.java:48)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"} {"timestamp":"2022-09-20T01:56:10.605Z","level":"WARN","thread":"Thread-5","logger":"org.forgerock.openam.services.push.PushNotificationService","message":"No Push Notification Delegate configured for realm /","context":"default"} {"timestamp":"2022-09-20T01:56:10.872Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.iplanet.services.naming.WebtopNaming","message":"WebtopNaming.configMonitoring: start generating ssoserverinfo\n Start time = 2022-09-20 01:56:10","context":"default"} {"timestamp":"2022-09-20T01:56:10.876Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringAdapters","message":"Agent.startAgent:Monitoring HTML port not enabled in configuration or JDMK not present.","context":"default"} {"timestamp":"2022-09-20T01:56:10.877Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringAdapters","message":"Agent.startAgent:Monitoring SNMP port not enabled or JDMK not present.","context":"default"} {"timestamp":"2022-09-20T01:56:10.877Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringAdapters","message":"Agent.startAgent:Monitoring RMI port not enabled.","context":"default"} {"timestamp":"2022-09-20T01:56:10.913Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.sm.ldap.SMSLdapObject","message":"SMSLdapObject:entryExists: LdapSmsEntryUid{dn='ou=default,ou=OrganizationConfig,ou=1.0,ou=sunFMCOTConfigService,ou=services,ou=am-config', dataStoreId=DataStoreId{id='application-store'}} does not exist","context":"default"} {"timestamp":"2022-09-20T01:56:10.923Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.sm.ldap.SMSLdapObject","message":"SMSLdapObject:entryExists: LdapSmsEntryUid{dn='ou=default,ou=OrganizationConfig,ou=1.0,ou=sunFMWSFederationMetadataService,ou=services,ou=am-config', dataStoreId=DataStoreId{id='application-store'}} does not exist","context":"default"} {"timestamp":"2022-09-20T01:56:10.931Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:56:10.937Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.log.s1is.LogConfigReader","message":"LogConfigReader: DB password is null","context":"default"} {"timestamp":"2022-09-20T01:56:10.937Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.log.s1is.LogConfigReader","message":"LogConfigReader: Logfile prefix string is null","context":"default"} {"timestamp":"2022-09-20T01:56:10.938Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.log.s1is.LogConfigReader","message":"LogConfigReader: Filter class is null","context":"default"} {"timestamp":"2022-09-20T01:56:10.944Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:56:10.964Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:56:10.969Z","level":"WARN","thread":"main","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-1"},"logger":"com.sun.identity.monitoring.FederationMonitoringSetupImpl","message":"JDMK runtime not found - Federation SNMP disabled","context":"default"} 20-Sep-2022 01:56:13.990 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/am] has finished in [22,640] ms 20-Sep-2022 01:56:13.994 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 20-Sep-2022 01:56:14.003 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8081"] 20-Sep-2022 01:56:14.005 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [22703] milliseconds {"timestamp":"2022-09-20T01:56:14.137Z","level":"WARN","thread":"http-nio-8080-exec-2","logger":"org.forgerock.openam.core.rest.docs.api.ApiDocsService","message":"API Descriptor has changed - regenerating docs","context":"default"} {"timestamp":"2022-09-20T01:56:14.499Z","level":"WARN","thread":"LDAP SDK Default AsyncRx I/O Thread (0)","logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ldapClient=LdapClient(host=ds-idrepo-1.ds-idrepo, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Connection refused","context":"default"} {"timestamp":"2022-09-20T01:56:14.824Z","level":"WARN","thread":"LDAP SDK Default AsyncRx I/O Thread (2)","logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=10, blocked=0, ldapClient=LdapClient(host=ds-idrepo-1.ds-idrepo, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Connection refused","context":"default"} {"timestamp":"2022-09-20T01:56:14.823Z","level":"ERROR","thread":"http-nio-8081-exec-1","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-17"},"logger":"org.forgerock.http.servlet.HttpFrameworkServlet","message":"Failed to write response - rootId:1445007e-b316-4513-8a5a-78a42412e6ab-21","context":"default","exception":"org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe\n\tat org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)\n\tat org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:784)\n\tat org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:689)\n\tat org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388)\n\tat org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366)\n\tat org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)\n\tat org.forgerock.http.io.IO.stream(IO.java:290)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.writeChfResponse(HttpFrameworkServlet.java:397)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.lambda$writeAndCloseChfResponse$2(HttpFrameworkServlet.java:359)\n\tat org.forgerock.http.servlet.Servlet3Adapter$SynchronousServletSynchronizer.runIfNotCompleted(Servlet3Adapter.java:151)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.writeAndCloseChfResponse(HttpFrameworkServlet.java:359)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.lambda$service$0(HttpFrameworkServlet.java:284)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenOnResult(Promises.java:131)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:284)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.forgerock.openam.http.OpenAMHttpFrameworkServlet.service(OpenAMHttpFrameworkServlet.java:47)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.services.datastore.DataStoreConsistencyFilter.lambda$doFilter$0(DataStoreConsistencyFilter.java:46)\n\tat org.forgerock.openam.service.datastore.ReentrantVolatileActionConsistencyController.safeExecute(ReentrantVolatileActionConsistencyController.java:37)\n\tat org.forgerock.openam.services.datastore.DataStoreConsistencyFilter.doFilter(DataStoreConsistencyFilter.java:46)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.rest.ProtocolVersionFilter.doFilter(ProtocolVersionFilter.java:63)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SecureCookieFilter.doFilter(SecureCookieFilter.java:63)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.DisableSameSiteCookiesFilter.doFilter(DisableSameSiteCookiesFilter.java:106)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:59)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:116)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.validation.RequestEntitySizeVerificationFilter.doFilter(RequestEntitySizeVerificationFilter.java:64)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:47)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: java.io.IOException: Broken pipe\n\tat java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)\n\tat java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)\n\tat java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)\n\tat java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:462)\n\tat org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135)\n\tat org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1424)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:768)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:593)\n\tat org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:537)\n\tat org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:547)\n\tat org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73)\n\tat org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:194)\n\tat org.apache.coyote.Response.doWrite(Response.java:615)\n\tat org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340)\n\t... 73 common frames omitted\n"} {"timestamp":"2022-09-20T01:56:15.841Z","level":"WARN","thread":"LDAP SDK Default Scheduler","logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=66, blocked=0, ldapClient=LdapClient(host=ds-cts-2.ds-cts, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Host 'ds-cts-2.ds-cts' cannot be resolved","context":"default"} {"timestamp":"2022-09-20T01:56:19.201Z","level":"WARN","thread":"LDAP SDK Default Scheduler","logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=2, blocked=0, ldapClient=LdapClient(host=ds-idrepo-2.ds-idrepo, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Host 'ds-idrepo-2.ds-idrepo' cannot be resolved","context":"default"} {"timestamp":"2022-09-20T01:56:19.201Z","level":"WARN","thread":"LDAP SDK Default AsyncRx I/O Thread (2)","logger":"org.forgerock.opendj.ldap.LoadBalancer","message":"Connection factory 'CachedConnectionPool(size=0[in:0 + out:0 + pending:0], maxSize=2, blocked=0, ldapClient=LdapClient(host=ds-idrepo-1.ds-idrepo, port=1636, protocol=LDAPS))' is no longer operational: Connect Error: Connection refused","context":"default"} 10.0.3.1 - - [20/Sep/2022:01:56:14 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 727ms 10.0.3.1 - - [20/Sep/2022:01:56:14 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 727ms 10.0.4.6 - - [20/Sep/2022:01:56:14 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 98304 793ms 10.0.3.1 - - [20/Sep/2022:01:56:20 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 3ms 10.0.4.6 - - [20/Sep/2022:01:56:27 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 199046 29ms 10.0.3.1 - - [20/Sep/2022:01:56:30 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 7ms 10.0.3.1 - - [20/Sep/2022:01:56:40 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms 10.0.3.1 - - [20/Sep/2022:01:56:50 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 4ms 10.0.3.1 - - [20/Sep/2022:01:56:50 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms 10.0.4.6 - - [20/Sep/2022:01:56:57 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 199048 27ms 10.0.3.1 - - [20/Sep/2022:01:57:00 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 7ms 10.0.3.1 - - [20/Sep/2022:01:57:10 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms 10.0.3.1 - - [20/Sep/2022:01:57:20 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 3ms 10.0.3.1 - - [20/Sep/2022:01:57:20 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms 10.0.4.6 - - [20/Sep/2022:01:57:27 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 199052 25ms 10.0.3.1 - - [20/Sep/2022:01:57:30 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms 10.0.3.1 - - [20/Sep/2022:01:57:40 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms {"timestamp":"2022-09-20T01:57:50.021Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"org.forgerock.openam.core.realms.impl.DefaultRealmLookup","message":"DefaultRealms:lookup Unable to find Org name for: authenticate","context":"default"} {"timestamp":"2022-09-20T01:57:50.022Z","level":"DEBUG","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2022-09-20T01:57:50.022Z","level":"DEBUG","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2022-09-20T01:57:50.217Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.authentication.config.AMAuthLevelManager","message":"No auth level for module OpenIdConnect","context":"default"} {"timestamp":"2022-09-20T01:57:50.218Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.authentication.config.AMAuthLevelManager","message":"No auth level for module AuthenticatorPushRegistration","context":"default"} {"timestamp":"2022-09-20T01:57:50.219Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.authentication.config.AMAuthLevelManager","message":"No auth level for module PersistentCookie","context":"default"} {"timestamp":"2022-09-20T01:57:50.270Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:57:50.366Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.idm.plugins.internal.AgentsRepo","message":"AgentsRepo.getAgentGroupConfig: Unable to get Agent Group Config due to The instance agentgroup does not exist","context":"default"} {"timestamp":"2022-09-20T01:57:50.437Z","eventName":"AM-LOGIN-MODULE-COMPLETED","transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76","trackingIds":["1445007e-b316-4513-8a5a-78a42412e6ab-85"],"userId":"id=amadmin,ou=user,ou=am-config","principal":["amadmin"],"entries":[{"moduleId":"DataStore","info":{"authControlFlag":"REQUIRED","moduleClass":"DataStore","ipAddress":"10.0.1.4","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"1445007e-b316-4513-8a5a-78a42412e6ab-87"} {"timestamp":"2022-09-20T01:57:50.447Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.idm.AMIdentity","message":"Error occurred attempting to instantiate an identity","context":"default","exception":"java.lang.IllegalArgumentException: Invalid identity DN \n\tat org.forgerock.am.identity.domain.Identity.of(Identity.java:153)\n\tat org.forgerock.am.identity.domain.Identity.of(Identity.java:139)\n\tat com.sun.identity.idm.AMIdentity.(AMIdentity.java:124)\n\tat com.sun.identity.idm.IdentityStoreImpl.searchForIdentity(IdentityStoreImpl.java:491)\n\tat com.sun.identity.authentication.service.AuthD.getIdentity(AuthD.java:1146)\n\tat com.sun.identity.authentication.service.LoginState.getUserUniversalId(LoginState.java:5564)\n\tat com.sun.identity.authentication.service.LoginState.searchUserProfile(LoginState.java:2432)\n\tat com.sun.identity.authentication.service.AMLoginContext.runLogin(AMLoginContext.java:555)\n\tat com.sun.identity.authentication.server.AuthContextLocal.submitRequirements(AuthContextLocal.java:591)\n\tat org.forgerock.openam.core.rest.authn.core.wrappers.AuthContextLocalWrapper.submitRequirements(AuthContextLocalWrapper.java:108)\n\tat org.forgerock.openam.core.rest.authn.core.LoginProcess.next(LoginProcess.java:169)\n\tat org.forgerock.openam.core.rest.authn.RestAuthenticationHandler.processAuthentication(RestAuthenticationHandler.java:540)\n\tat org.forgerock.openam.core.rest.authn.RestAuthenticationHandler.authenticate(RestAuthenticationHandler.java:266)\n\tat org.forgerock.openam.core.rest.authn.http.AuthenticationServiceV1.authenticate(AuthenticationServiceV1.java:157)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.forgerock.openam.http.annotations.AnnotatedMethod.invoke(AnnotatedMethod.java:81)\n\tat org.forgerock.openam.http.annotations.Endpoints$1.handle(Endpoints.java:77)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.rest.DisableCachingFilter.filter(DisableCachingFilter.java:90)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.audit.AbstractHttpAccessAuditFilter.filter(AbstractHttpAccessAuditFilter.java:88)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.cors.CorsFilter.filter(CorsFilter.java:91)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.rest.RealmContextFilter.filter(RealmContextFilter.java:87)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.rest.CsrfFilter.filter(CsrfFilter.java:91)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.routing.ResourceApiVersionRoutingFilter.filter(ResourceApiVersionRoutingFilter.java:59)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.grantAccess(AuthenticationFramework.java:188)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.lambda$onValidateRequestSuccess$1(AuthenticationFramework.java:181)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.validateRequest(AuthenticationFramework.java:144)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.processMessage(AuthenticationFramework.java:134)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFilter.filter(AuthenticationFilter.java:84)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.GuiceHandler.handle(GuiceHandler.java:51)\n\tat org.forgerock.openam.http.HttpRoute$6.handle(HttpRoute.java:206)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.dpro.session.ProofOfPossessionTokenFilter.filter(ProofOfPossessionTokenFilter.java:95)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.swagger.OpenApiRequestFilter.filter(OpenApiRequestFilter.java:63)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.ApiDescriptorFilter.filter(ApiDescriptorFilter.java:58)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.ResponseContext$ResponseContextFilter.filter(ResponseContext.java:53)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.OpenAMHttpApplication.lambda$static$1(OpenAMHttpApplication.java:60)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.OpenAMHttpApplication.lambda$cacheHeaderFilter$3(OpenAMHttpApplication.java:88)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:86)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:282)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.forgerock.openam.http.OpenAMHttpFrameworkServlet.service(OpenAMHttpFrameworkServlet.java:47)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.services.datastore.DataStoreConsistencyFilter.lambda$doFilter$0(DataStoreConsistencyFilter.java:46)\n\tat org.forgerock.openam.service.datastore.ReentrantVolatileActionConsistencyController.safeExecute(ReentrantVolatileActionConsistencyController.java:37)\n\tat org.forgerock.openam.services.datastore.DataStoreConsistencyFilter.doFilter(DataStoreConsistencyFilter.java:46)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.rest.ProtocolVersionFilter.doFilter(ProtocolVersionFilter.java:66)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SecureCookieFilter.doFilter(SecureCookieFilter.java:63)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.DisableSameSiteCookiesFilter.doFilter(DisableSameSiteCookiesFilter.java:106)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:59)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:116)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.validation.RequestEntitySizeVerificationFilter.doFilter(RequestEntitySizeVerificationFilter.java:64)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:47)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"} {"timestamp":"2022-09-20T01:57:50.457Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.idm.AMIdentity","message":"Error occurred attempting to instantiate an identity","context":"default","exception":"java.lang.IllegalArgumentException: Invalid identity DN \n\tat org.forgerock.am.identity.domain.Identity.of(Identity.java:153)\n\tat org.forgerock.am.identity.domain.Identity.of(Identity.java:139)\n\tat com.sun.identity.idm.AMIdentity.(AMIdentity.java:124)\n\tat com.sun.identity.authentication.service.LoginState.getUserProfile(LoginState.java:2170)\n\tat com.sun.identity.authentication.service.LoginState.getUserProfile(LoginState.java:2116)\n\tat com.sun.identity.authentication.service.LoginState.getCreateUserProfile(LoginState.java:2590)\n\tat com.sun.identity.authentication.service.LoginState.searchUserProfile(LoginState.java:2437)\n\tat com.sun.identity.authentication.service.AMLoginContext.runLogin(AMLoginContext.java:555)\n\tat com.sun.identity.authentication.server.AuthContextLocal.submitRequirements(AuthContextLocal.java:591)\n\tat org.forgerock.openam.core.rest.authn.core.wrappers.AuthContextLocalWrapper.submitRequirements(AuthContextLocalWrapper.java:108)\n\tat org.forgerock.openam.core.rest.authn.core.LoginProcess.next(LoginProcess.java:169)\n\tat org.forgerock.openam.core.rest.authn.RestAuthenticationHandler.processAuthentication(RestAuthenticationHandler.java:540)\n\tat org.forgerock.openam.core.rest.authn.RestAuthenticationHandler.authenticate(RestAuthenticationHandler.java:266)\n\tat org.forgerock.openam.core.rest.authn.http.AuthenticationServiceV1.authenticate(AuthenticationServiceV1.java:157)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.forgerock.openam.http.annotations.AnnotatedMethod.invoke(AnnotatedMethod.java:81)\n\tat org.forgerock.openam.http.annotations.Endpoints$1.handle(Endpoints.java:77)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.rest.DisableCachingFilter.filter(DisableCachingFilter.java:90)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.audit.AbstractHttpAccessAuditFilter.filter(AbstractHttpAccessAuditFilter.java:88)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.cors.CorsFilter.filter(CorsFilter.java:91)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.rest.RealmContextFilter.filter(RealmContextFilter.java:87)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.rest.CsrfFilter.filter(CsrfFilter.java:91)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.routing.ResourceApiVersionRoutingFilter.filter(ResourceApiVersionRoutingFilter.java:59)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.grantAccess(AuthenticationFramework.java:188)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.lambda$onValidateRequestSuccess$1(AuthenticationFramework.java:181)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:263)\n\tat org.forgerock.util.promise.Promises$CompletedPromise.thenAsync(Promises.java:252)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.validateRequest(AuthenticationFramework.java:144)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFramework.processMessage(AuthenticationFramework.java:134)\n\tat org.forgerock.caf.authentication.framework.AuthenticationFilter.filter(AuthenticationFilter.java:84)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.GuiceHandler.handle(GuiceHandler.java:51)\n\tat org.forgerock.openam.http.HttpRoute$6.handle(HttpRoute.java:206)\n\tat org.forgerock.http.routing.Router.handle(Router.java:100)\n\tat org.forgerock.openam.dpro.session.ProofOfPossessionTokenFilter.filter(ProofOfPossessionTokenFilter.java:95)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.swagger.OpenApiRequestFilter.filter(OpenApiRequestFilter.java:63)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.ApiDescriptorFilter.filter(ApiDescriptorFilter.java:58)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.ResponseContext$ResponseContextFilter.filter(ResponseContext.java:53)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.OpenAMHttpApplication.lambda$static$1(OpenAMHttpApplication.java:60)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.openam.http.OpenAMHttpApplication.lambda$cacheHeaderFilter$3(OpenAMHttpApplication.java:88)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.filter.TransactionIdInboundFilter.filter(TransactionIdInboundFilter.java:86)\n\tat org.forgerock.http.handler.Handlers$1.handle(Handlers.java:54)\n\tat org.forgerock.http.servlet.HttpFrameworkServlet.service(HttpFrameworkServlet.java:282)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.forgerock.openam.http.OpenAMHttpFrameworkServlet.service(OpenAMHttpFrameworkServlet.java:47)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:764)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.services.datastore.DataStoreConsistencyFilter.lambda$doFilter$0(DataStoreConsistencyFilter.java:46)\n\tat org.forgerock.openam.service.datastore.ReentrantVolatileActionConsistencyController.safeExecute(ReentrantVolatileActionConsistencyController.java:37)\n\tat org.forgerock.openam.services.datastore.DataStoreConsistencyFilter.doFilter(DataStoreConsistencyFilter.java:46)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.rest.ProtocolVersionFilter.doFilter(ProtocolVersionFilter.java:66)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SecureCookieFilter.doFilter(SecureCookieFilter.java:63)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.DisableSameSiteCookiesFilter.doFilter(DisableSameSiteCookiesFilter.java:106)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.validation.ResponseValidationFilter.doFilter(ResponseValidationFilter.java:59)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.headers.SetHeadersFilter.doFilter(SetHeadersFilter.java:110)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat com.sun.identity.setup.AMSetupFilter.doFilter(AMSetupFilter.java:116)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.validation.RequestEntitySizeVerificationFilter.doFilter(RequestEntitySizeVerificationFilter.java:64)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.forgerock.openam.audit.context.AuditContextFilter.doFilter(AuditContextFilter.java:47)\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\tat org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"} {"timestamp":"2022-09-20T01:57:50.617Z","level":"WARN","thread":"http-nio-8081-exec-8","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76"},"logger":"com.sun.identity.monitoring.MonitoringServicesImpl","message":"JDMK runtime not found - Policy Monitoring disabled","context":"default"} {"timestamp":"2022-09-20T01:57:50.607Z","eventName":"AM-SESSION-CREATED","transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76","trackingIds":["1445007e-b316-4513-8a5a-78a42412e6ab-85"],"userId":"id=amadmin,ou=user,ou=am-config","runAs":"id=amadmin,ou=user,ou=am-config","objectId":"1445007e-b316-4513-8a5a-78a42412e6ab-85","operation":"CREATE","realm":"/","component":"Session","source":"audit","topic":"activity","level":"INFO","_eventId":"1445007e-b316-4513-8a5a-78a42412e6ab-92"} {"timestamp":"2022-09-20T01:57:50.631Z","eventName":"AM-LOGIN-COMPLETED","transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76","trackingIds":["1445007e-b316-4513-8a5a-78a42412e6ab-85"],"userId":"id=amadmin,ou=user,ou=am-config","principal":["amadmin"],"entries":[{"moduleId":"DataStore","info":{"ipAddress":"10.0.1.4","authLevel":"0"}}],"result":"SUCCESSFUL","realm":"/","component":"Authentication","source":"audit","topic":"authentication","level":"INFO","_eventId":"1445007e-b316-4513-8a5a-78a42412e6ab-96"} {"timestamp":"2022-09-20T01:57:50.652Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-76","trackingIds":["1445007e-b316-4513-8a5a-78a42412e6ab-85"],"userId":"id=amadmin,ou=user,ou=am-config","client":{"ip":"10.0.1.4","port":52406},"http":{"request":{"secure":true,"method":"POST","path":"https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/authenticate","headers":{"accept":["*/*"],"accept-api-version":["protocol=1.0,resource=2.0"],"content-type":["application/json"],"host":["xlou.iam.xlou-cdm.engineeringpit.com"],"user-agent":["python-requests/2.28.1"],"x-forwarded-for":["35.212.145.225"],"x-forwarded-host":["xlou.iam.xlou-cdm.engineeringpit.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["35.212.145.225"],"x-request-id":["403bba885a56cb6a05eb69e932f74a68"],"x-scheme":["https"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":699,"elapsedTimeUnits":"MILLISECONDS"},"realm":"/","component":"Authentication","source":"audit","topic":"access","level":"INFO","_eventId":"1445007e-b316-4513-8a5a-78a42412e6ab-102"} {"timestamp":"2022-09-20T01:57:50.901Z","level":"WARN","thread":"http-nio-8081-exec-9","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-103"},"logger":"org.forgerock.openam.core.realms.impl.DefaultRealmLookup","message":"DefaultRealms:lookup Unable to find Org name for: serverinfo","context":"default"} {"timestamp":"2022-09-20T01:57:50.902Z","level":"DEBUG","thread":"http-nio-8081-exec-9","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-103"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2022-09-20T01:57:50.902Z","level":"DEBUG","thread":"http-nio-8081-exec-9","mdc":{"transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-103"},"logger":"org.forgerock.openam.cors.CorsFilter","message":"handling non-CORS request","context":"default"} {"timestamp":"2022-09-20T01:57:50.928Z","eventName":"AM-ACCESS-OUTCOME","transactionId":"1445007e-b316-4513-8a5a-78a42412e6ab-103","trackingIds":["1445007e-b316-4513-8a5a-78a42412e6ab-85"],"userId":"id=amadmin,ou=user,ou=am-config","client":{"ip":"10.0.1.4","port":52412},"server":{"ip":"10.0.3.10","port":8081},"http":{"request":{"secure":true,"method":"GET","path":"https://xlou.iam.xlou-cdm.engineeringpit.com/am/json/serverinfo/version","headers":{"accept":["*/*"],"accept-api-version":["protocol=2.1"],"host":["xlou.iam.xlou-cdm.engineeringpit.com"],"user-agent":["python-requests/2.28.1"],"x-forwarded-for":["35.212.145.225"],"x-forwarded-host":["xlou.iam.xlou-cdm.engineeringpit.com"],"x-forwarded-port":["443"],"x-forwarded-proto":["https"],"x-real-ip":["35.212.145.225"],"x-request-id":["f314e596b9a839ddfbcfd8d6ba917737"],"x-scheme":["https"]}}},"request":{"protocol":"CREST","operation":"READ"},"response":{"status":"SUCCESSFUL","statusCode":"","elapsedTime":35,"elapsedTimeUnits":"MILLISECONDS","detail":{"revision":"-59589352"}},"realm":"/","component":"Server Info","source":"audit","topic":"access","level":"INFO","_eventId":"1445007e-b316-4513-8a5a-78a42412e6ab-107"} 10.0.3.1 - - [20/Sep/2022:01:57:50 +0000] "GET /am/json/health/live HTTP/1.1" 200 - 3ms 10.0.3.1 - - [20/Sep/2022:01:57:50 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms 10.0.1.4 - - [20/Sep/2022:01:57:50 +0000] "POST /am/json/authenticate?realm=/ HTTP/1.1" 200 167 707ms 10.0.1.4 - - [20/Sep/2022:01:57:50 +0000] "GET /am/json/serverinfo/version HTTP/1.1" 200 282 47ms 10.0.4.6 - - [20/Sep/2022:01:57:57 +0000] "GET /am/json/metrics/prometheus HTTP/1.1" 200 355655 55ms 10.0.3.1 - - [20/Sep/2022:01:58:00 +0000] "GET /am/json/health/ready HTTP/1.1" 200 - 5ms