--Task-- name: CheckReplication_ds-idrepo enabled: True class_name: CheckReplicationTask source_name: ds-idrepo source_namespace: >default< target_name: ds-idrepo target_namespace: >default< start: 0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] options: {} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [04/Feb/2023 03:57:41] CheckReplication_ds-idrepo pre : N/A ________________________________________________________________________________ max_time: 13h 59m sleep_interval: 10 [04/Feb/2023 03:57:41] - INFO: Timeout for this Task has changed to 1d 04h (100800 seconds) [04/Feb/2023 03:57:41] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [04/Feb/2023 03:57:41] CheckReplication_ds-idrepo step1 : Check number of entries ________________________________________________________________________________ loop 1 : ERROR : Pod "ds-idrepo-0" has 1000021 entries but pod "ds-idrepo-1" has 1000022 entries (should have same number of entries) loop 1 : ERROR : Pod "ds-idrepo-0" has 1000021 entries but pod "ds-idrepo-2" has 1000022 entries (should have same number of entries) loop 1 : SUCCESS: Pods "ds-idrepo-1" and "ds-idrepo-2" are in sync (both have 1000022 entries) loop 1 : Sleep 10 and retry loop 2 : SUCCESS: Pods "ds-idrepo-0" and "ds-idrepo-1" are in sync (both have 1000021 entries) loop 2 : SUCCESS: Pods "ds-idrepo-0" and "ds-idrepo-2" are in sync (both have 1000021 entries) loop 2 : SUCCESS: Pods "ds-idrepo-1" and "ds-idrepo-2" are in sync (both have 1000021 entries) [check_number_of_entries] SUCCESS : replication is synchronised after 15s ________________________________________________________________________________ [04/Feb/2023 03:58:02] CheckReplication_ds-idrepo step2 : Monitor replication attributes in cn=monitor ________________________________________________________________________________ loop 1 : SUCCESS: ds-idrepo-0 : ds-mon-current-delay = 0 loop 1 : SUCCESS: ds-idrepo-0 : ds-mon-updates-inbound-queue = 0 loop 1 : SUCCESS: ds-idrepo-1 : ds-mon-current-delay = 0 loop 1 : SUCCESS: ds-idrepo-1 : ds-mon-updates-inbound-queue = 0 loop 1 : SUCCESS: ds-idrepo-2 : ds-mon-current-delay = 0 loop 1 : SUCCESS: ds-idrepo-2 : ds-mon-updates-inbound-queue = 0 [check_monitoring_replication] final status : { "ds-idrepo-0": { "status": true, "ds-mon-current-delay": 0, "ds-mon-updates-inbound-queue": 0 }, "ds-idrepo-1": { "status": true, "ds-mon-current-delay": 0, "ds-mon-updates-inbound-queue": 0 }, "ds-idrepo-2": { "status": true, "ds-mon-current-delay": 0, "ds-mon-updates-inbound-queue": 0 } } [check_monitoring_replication] SUCCESS : replication data in cn=monitor is synchronised after 0s ________________________________________________________________________________ [04/Feb/2023 03:58:07] CheckReplication_ds-idrepo step3 : Temporary step for PERF-1831 : Dump cn=monitor replication attributes ________________________________________________________________________________ ---------------- Pod ds-idrepo-0 ---------------- [run_command]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "cn=replicas,cn=replication,cn=monitor" "objectclass=*" [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- [print_head_tail]: Print head (80 lines) and tail (80 lines) of input string (172 lines): ----- output ----- dn: cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: replicas dn: ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.4:8989 ds-mon-connected-to-server-id: ds-idrepo-0 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 555879 ds-mon-domain-name: dc=openidm,dc=forgerock,dc=io ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.000,"p99999":0.000} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 409 ds-mon-server-id: ds-idrepo-0 ds-mon-server-state: 010b01861a92ff6f000f55b0ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034042.477Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [3, 1, 1, 0, 2, 0, 0, 1, 0, 0, 1, 0, 2, 0, 0, 2] dn: ds-mon-domain-name=ou=am-config,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.4:8989 ds-mon-connected-to-server-id: ds-idrepo-0 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 2338230 ds-mon-domain-name: ou=am-config ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":13,"total":58.348,"mean_rate":0.012,"m1_rate":0.000,"m5_rate":0.002,"m15_rate":0.005,"mean":4.490,"min":1.966,"max":10.813,"stddev":2.082,"p50":3.965,"p75":4.751,"p95":10.813,"p98":10.813,"p99":10.813,"p999":10.813,"p9999":10.813,"p99999":10.813} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 2 ds-mon-server-id: ds-idrepo-0 ds-mon-server-state: 010b01861a83259d00000022ds-idrepo-0 ds-mon-server-state: 010b01861a84b4b8000000aads-idrepo-2 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034042.766Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [3, 1, 1, 0, 2, 0, 0, 1, 0, 0, 1, 0, 2, 0, 0, 2] dn: cn=remote replicas,ds-mon-domain-name=ou=am-config,cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: remote replicas dn: ds-mon-server-id=ds-idrepo-2,cn=remote replicas,ds-mon-domain-name=ou=am-config,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-remote-replica ds-mon-domain-name: ou=am-config ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 [...] ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.000,"p99999":0.000} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 1000005 ds-mon-server-id: ds-idrepo-0 ds-mon-server-state: 010b01861a92b396000f5548ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034042.830Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [3, 1, 1, 0, 2, 0, 0, 1, 0, 0, 1, 0, 2, 0, 0, 2] dn: ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.4:8989 ds-mon-connected-to-server-id: ds-idrepo-0 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 69125 ds-mon-domain-name: ou=tokens ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.000,"p99999":0.000} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 2 ds-mon-server-id: ds-idrepo-0 ds-mon-server-state: 010b01861a831b7200000011ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034042.885Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [3, 1, 1, 0, 2, 0, 0, 1, 0, 0, 1, 0, 2, 0, 0, 2] dn: ds-mon-domain-name=uid=monitor,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.4:8989 ds-mon-connected-to-server-id: ds-idrepo-0 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 48 ds-mon-domain-name: uid=monitor ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.000,"p99999":0.000} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034042.710Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [3, 1, 1, 0, 2, 0, 0, 1, 0, 0, 1, 0, 2, 0, 0, 2] --- stderr --- ---------------- Pod ds-idrepo-1 ---------------- [run_command]: kubectl --namespace=xlou exec ds-idrepo-1 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "cn=replicas,cn=replication,cn=monitor" "objectclass=*" [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- [print_head_tail]: Print head (80 lines) and tail (80 lines) of input string (230 lines): ----- output ----- dn: cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: replicas dn: ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.132:8989 ds-mon-connected-to-server-id: ds-idrepo-1 ds-mon-current-receive-window: 2147483314 ds-mon-domain-generation-id: 555879 ds-mon-domain-name: dc=openidm,dc=forgerock,dc=io ds-mon-entries-awaiting-updates-count: 1 ds-mon-lost-connections: 0 ds-mon-receive-delay: 187 ds-mon-replay-delay: 801 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":411,"total":500.644,"mean_rate":0.407,"m1_rate":0.477,"m5_rate":0.408,"m15_rate":0.277,"mean":1.218,"min":0.518,"max":13.959,"stddev":1.298,"p50":0.770,"p75":1.122,"p95":2.802,"p98":5.276,"p99":6.619,"p999":13.959,"p9999":13.959,"p99999":13.959} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-1 ds-mon-server-state: 010b01861a930fdc000f55c6ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034120.702Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 1 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [62419, 61971, 62419, 62112, 62617, 62641, 62821, 62716, 62502, 62597, 62572, 62778, 62624, 63107, 61847, 62690] dn: cn=remote replicas,ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: remote replicas dn: ds-mon-server-id=ds-idrepo-0,cn=remote replicas,ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-remote-replica ds-mon-domain-name: dc=openidm,dc=forgerock,dc=io ds-mon-receive-delay: 187 ds-mon-replay-delay: 801 ds-mon-replayed-updates: {"count":411,"total":500.644,"mean_rate":0.425,"m1_rate":0.477,"m5_rate":0.552,"m15_rate":1.509,"mean":1.218,"min":0.518,"max":13.959,"stddev":1.298,"p50":0.770,"p75":1.122,"p95":2.802,"p98":5.276,"p99":6.619,"p999":13.959,"p9999":13.959,"p99999":13.959} ds-mon-server-id: ds-idrepo-0 dn: ds-mon-domain-name=ou=am-config,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.132:8989 ds-mon-connected-to-server-id: ds-idrepo-1 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 2338230 ds-mon-domain-name: ou=am-config ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":15,"total":76.671,"mean_rate":0.015,"m1_rate":0.000,"m5_rate":0.016,"m15_rate":0.136,"mean":5.113,"min":2.212,"max":14.680,"stddev":3.155,"p50":4.194,"p75":5.407,"p95":14.680,"p98":14.680,"p99":14.680,"p999":14.680,"p9999":14.680,"p99999":14.680} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-1 ds-mon-server-state: 010b01861a83259d00000022ds-idrepo-0 ds-mon-server-state: 010b01861a84b4b8000000aads-idrepo-2 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034121.079Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 [...] ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-updates: {"count":1000005,"total":5008736.075,"mean_rate":989.569,"m1_rate":109.068,"m5_rate":819.067,"m15_rate":670.470,"mean":5.009,"min":0.778,"max":205.521,"stddev":5.008,"p50":3.785,"p75":6.160,"p95":12.321,"p98":16.777,"p99":21.234,"p999":64.487,"p9999":110.100,"p99999":164.626} ds-mon-server-id: ds-idrepo-0 dn: ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.132:8989 ds-mon-connected-to-server-id: ds-idrepo-1 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 69125 ds-mon-domain-name: ou=tokens ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":2,"total":17.108,"mean_rate":0.002,"m1_rate":0.000,"m5_rate":0.014,"m15_rate":0.131,"mean":8.540,"min":2.556,"max":14.549,"stddev":5.976,"p50":2.572,"p75":14.549,"p95":14.549,"p98":14.549,"p99":14.549,"p999":14.549,"p9999":14.549,"p99999":14.549} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-1 ds-mon-server-state: 010b01861a831b7200000011ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034121.265Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [62419, 61971, 62419, 62112, 62617, 62641, 62821, 62716, 62502, 62597, 62572, 62778, 62624, 63107, 61847, 62690] dn: cn=remote replicas,ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: remote replicas dn: ds-mon-server-id=ds-idrepo-0,cn=remote replicas,ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-remote-replica ds-mon-domain-name: ou=tokens ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-updates: {"count":2,"total":17.108,"mean_rate":0.002,"m1_rate":0.000,"m5_rate":0.014,"m15_rate":0.131,"mean":8.540,"min":2.556,"max":14.549,"stddev":5.976,"p50":2.572,"p75":14.549,"p95":14.549,"p98":14.549,"p99":14.549,"p999":14.549,"p9999":14.549,"p99999":14.549} ds-mon-server-id: ds-idrepo-0 dn: ds-mon-domain-name=uid=monitor,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.52.132:8989 ds-mon-connected-to-server-id: ds-idrepo-1 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 48 ds-mon-domain-name: uid=monitor ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.000,"p99999":0.000} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-1 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034120.990Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [62419, 61971, 62419, 62112, 62617, 62641, 62821, 62716, 62502, 62597, 62572, 62778, 62624, 63107, 61847, 62690] --- stderr --- ---------------- Pod ds-idrepo-2 ---------------- [run_command]: kubectl --namespace=xlou exec ds-idrepo-2 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "cn=replicas,cn=replication,cn=monitor" "objectclass=*" [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- [print_head_tail]: Print head (80 lines) and tail (80 lines) of input string (220 lines): ----- output ----- dn: cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: replicas dn: ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.54.132:8989 ds-mon-connected-to-server-id: ds-idrepo-2 ds-mon-current-receive-window: 2147483315 ds-mon-domain-generation-id: 555879 ds-mon-domain-name: dc=openidm,dc=forgerock,dc=io ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":411,"total":488.203,"mean_rate":0.422,"m1_rate":0.477,"m5_rate":0.408,"m15_rate":0.277,"mean":1.188,"min":0.520,"max":14.549,"stddev":1.277,"p50":0.758,"p75":1.122,"p95":2.769,"p98":4.948,"p99":5.767,"p999":14.549,"p9999":14.549,"p99999":14.549} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-2 ds-mon-server-state: 010b01861a9312fd000f55cads-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034158.539Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [62416, 61970, 62418, 62112, 62615, 62641, 62820, 62715, 62502, 62597, 62571, 62778, 62622, 63107, 61847, 62688] dn: cn=remote replicas,ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: remote replicas dn: ds-mon-server-id=ds-idrepo-0,cn=remote replicas,ds-mon-domain-name=dc=openidm\,dc=forgerock\,dc=io,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-remote-replica ds-mon-domain-name: dc=openidm,dc=forgerock,dc=io ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-updates: {"count":411,"total":488.203,"mean_rate":0.424,"m1_rate":0.477,"m5_rate":0.552,"m15_rate":1.509,"mean":1.188,"min":0.520,"max":14.549,"stddev":1.277,"p50":0.758,"p75":1.122,"p95":2.769,"p98":4.948,"p99":5.767,"p999":14.549,"p9999":14.549,"p99999":14.549} ds-mon-server-id: ds-idrepo-0 dn: ds-mon-domain-name=ou=am-config,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.54.132:8989 ds-mon-connected-to-server-id: ds-idrepo-2 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 2338230 ds-mon-domain-name: ou=am-config ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":2,"total":22.801,"mean_rate":0.002,"m1_rate":0.000,"m5_rate":0.016,"m15_rate":0.137,"mean":11.428,"min":4.686,"max":18.219,"stddev":6.726,"p50":4.719,"p75":18.219,"p95":18.219,"p98":18.219,"p99":18.219,"p999":18.219,"p9999":18.219,"p99999":18.219} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 13 ds-mon-server-id: ds-idrepo-2 ds-mon-server-state: 010b01861a84b4b8000000aads-idrepo-2 ds-mon-server-state: 010b01861a83259d00000022ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034159.074Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 [...] ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-updates: {"count":1000005,"total":5094787.970,"mean_rate":1026.412,"m1_rate":112.781,"m5_rate":824.665,"m15_rate":672.004,"mean":5.095,"min":0.782,"max":198.181,"stddev":4.910,"p50":3.883,"p75":6.324,"p95":12.517,"p98":17.039,"p99":21.234,"p999":61.342,"p9999":113.246,"p99999":138.412} ds-mon-server-id: ds-idrepo-0 dn: ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.54.132:8989 ds-mon-connected-to-server-id: ds-idrepo-2 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 69125 ds-mon-domain-name: ou=tokens ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":2,"total":20.546,"mean_rate":0.002,"m1_rate":0.000,"m5_rate":0.016,"m15_rate":0.137,"mean":10.285,"min":2.540,"max":18.088,"stddev":7.737,"p50":2.556,"p75":18.088,"p95":18.088,"p98":18.088,"p99":18.088,"p999":18.088,"p9999":18.088,"p99999":18.088} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-2 ds-mon-server-state: 010b01861a831b7200000011ds-idrepo-0 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034159.229Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [62416, 61970, 62418, 62112, 62615, 62641, 62820, 62715, 62502, 62597, 62571, 62778, 62622, 63107, 61847, 62688] dn: cn=remote replicas,ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectclass: top objectclass: ds-monitor objectclass: ds-monitor-branch cn: remote replicas dn: ds-mon-server-id=ds-idrepo-0,cn=remote replicas,ds-mon-domain-name=ou=tokens,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-remote-replica ds-mon-domain-name: ou=tokens ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-updates: {"count":2,"total":20.546,"mean_rate":0.002,"m1_rate":0.000,"m5_rate":0.016,"m15_rate":0.137,"mean":10.285,"min":2.540,"max":18.088,"stddev":7.737,"p50":2.556,"p75":18.088,"p95":18.088,"p98":18.088,"p99":18.088,"p999":18.088,"p9999":18.088,"p99999":18.088} ds-mon-server-id: ds-idrepo-0 dn: ds-mon-domain-name=uid=monitor,cn=replicas,cn=replication,cn=monitor objectClass: top objectClass: ds-monitor objectClass: ds-monitor-replica ds-mon-connected-to-server-hostport: 10.56.54.132:8989 ds-mon-connected-to-server-id: ds-idrepo-2 ds-mon-current-receive-window: 2147483647 ds-mon-domain-generation-id: 48 ds-mon-domain-name: uid=monitor ds-mon-entries-awaiting-updates-count: 0 ds-mon-lost-connections: 0 ds-mon-receive-delay: 0 ds-mon-replay-delay: 0 ds-mon-replayed-internal-updates: 0 ds-mon-replayed-updates: {"count":0,"total":0.000,"mean_rate":0.000,"m1_rate":0.000,"m5_rate":0.000,"m15_rate":0.000,"mean":0.000,"min":0.000,"max":0.000,"stddev":0.000,"p50":0.000,"p75":0.000,"p95":0.000,"p98":0.000,"p99":0.000,"p999":0.000,"p9999":0.000,"p99999":0.000} ds-mon-replayed-updates-conflicts-resolved: 0 ds-mon-replayed-updates-conflicts-unresolved: 0 ds-mon-sent-updates: 0 ds-mon-server-id: ds-idrepo-2 ds-mon-ssl-encryption: true ds-mon-status: Normal ds-mon-status-last-changed: 20230204034158.848Z ds-mon-total-update-entry-count: 0 ds-mon-total-update-entry-left: 0 ds-mon-updates-already-in-progress: 0 ds-mon-updates-inbound-queue: 0 ds-mon-updates-outbound-queue: 0 ds-mon-updates-totals-per-replay-thread: [62416, 61970, 62418, 62112, 62615, 62641, 62820, 62715, 62502, 62597, 62571, 62778, 62622, 63107, 61847, 62688] --- stderr --- ________________________________________________________________________________ [04/Feb/2023 03:58:13] CheckReplication_ds-idrepo step4 : Temporary step for PERF-1831 : checks logs ________________________________________________________________________________ ---------------- Pod ds-idrepo-0 ---------------- [loop_until]: kubectl --namespace=xlou logs ds-idrepo-0 [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Initializing "data/changelogDb" from Docker image Initializing "data/import-tmp" from Docker image Upgrading configuration and data... * OpenDJ data has already been upgraded to version 7.3.0.35060cde0333c76dfba29a4e88896b885a0deba7 Rebuilding degraded indexes for base DN "ou=tokens"... Rebuilding degraded indexes for base DN "ou=identities"... Rebuilding degraded indexes for base DN "ou=am-config"... Rebuilding degraded indexes for base DN "dc=openidm,dc=forgerock,dc=io"... Server configured with: Group ID : default Server ID : ds-idrepo-0 Advertised listen address : ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local Bootstrap replication server(s) : ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8989,ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8989 {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:40.892Z","category":"CORE","severity":"NOTICE","msgId":134,"message":"ForgeRock Directory Services 7.3.0-20230127121641-35060cde0333c76dfba29a4e88896b885a0deba7 (build 20230127141714, revision number 35060cde0333c76dfba29a4e88896b885a0deba7) starting up"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:40.892Z","category":"JVM","severity":"NOTICE","msgId":21,"message":"Installation Directory: /opt/opendj"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:40.892Z","category":"JVM","severity":"NOTICE","msgId":23,"message":"Instance Directory: /opt/opendj"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:40.892Z","category":"JVM","severity":"NOTICE","msgId":17,"message":"JVM Information: 17.0.6+10-LTS by Azul Systems, Inc., 64-bit architecture, 11274289152 bytes heap size"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:40.892Z","category":"JVM","severity":"NOTICE","msgId":20,"message":"JVM Host: ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local default/ds-idrepo-0, running Linux 5.10.147+ amd64, unknown physical memory size, number of processors available 16"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:40.892Z","category":"JVM","severity":"NOTICE","msgId":19,"message":"JVM Arguments: \"-XX:+UseG1GC\", \"-XX:MaxRAMPercentage=75\", \"-XX:MaxTenuringThreshold=1\", \"-Djava.security.egd=file:/dev/urandom\", \"-Dorg.opends.server.scriptName=start-ds\""} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:41.493Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend amCts containing 5 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:41.693Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend idmRepo containing 42 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:41.894Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend cfgStore containing 115 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.094Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend amIdentityStore containing 21 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.294Z","category":"SYNC","severity":"NOTICE","msgId":204,"message":"Replication server RS(ds-idrepo-0) started listening for new connections on address 0.0.0.0 port 8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.494Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 42 entries and skipped 0 in 0 seconds (average rate 2333.3/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.494Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from directory server DS(ds-idrepo-0) for domain \"dc=openidm,dc=forgerock,dc=io\" at /10.56.52.4:60840"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-0) for domain \"dc=openidm,dc=forgerock,dc=io\" at 10.56.52.4:8989 with generation ID 555879"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from directory server DS(ds-idrepo-0) for domain \"uid=monitor\" at /10.56.52.4:60870"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-0) for domain \"uid=monitor\" at 10.56.52.4:8989 with generation ID 48"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 115 entries and skipped 0 in 0 seconds (average rate 8214.3/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from directory server DS(ds-idrepo-0) for domain \"ou=am-config\" at /10.56.52.4:60886"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-0) for domain \"ou=am-config\" at 10.56.52.4:8989 with generation ID 2338230"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 21 entries and skipped 0 in 0 seconds (average rate 7000.0/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from directory server DS(ds-idrepo-0) for domain \"ou=identities\" at /10.56.52.4:60898"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-0) for domain \"ou=identities\" at 10.56.52.4:8989 with generation ID 253127"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.694Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 5 entries and skipped 0 in 0 seconds (average rate 2500.0/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from directory server DS(ds-idrepo-0) for domain \"ou=tokens\" at /10.56.52.4:60924"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-0) for domain \"ou=tokens\" at 10.56.52.4:8989 with generation ID 69125"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"CORE","severity":"NOTICE","msgId":818,"message":"Registered 0 static groups, 0 dynamic groups and 0 virtual static groups. The static group cache is using 0 bytes of memory"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on Administration Connector 0.0.0.0:4444"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on LDAP 0.0.0.0:1389"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on HTTP 0.0.0.0:8080"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on HTTPS 0.0.0.0:8443"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on LDAPS 0.0.0.0:1636"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"CORE","severity":"NOTICE","msgId":135,"message":"The Directory Server has started successfully"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:40:42.895Z","category":"CORE","severity":"NOTICE","msgId":139,"message":"The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Directory Server has started successfully"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-3","timestamp":"2023-02-04T03:40:44.007Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-2","client":{"ip":"10.56.51.9","port":57286},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":28,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-25","timestamp":"2023-02-04T03:41:12.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-24","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":96,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.734Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"dc=openidm,dc=forgerock,dc=io\" at /10.56.52.132:49940"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.934Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"uid=monitor\" at /10.56.52.132:49948"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.134Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"ou=am-config\" at /10.56.52.132:49974"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.134Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"ou=identities\" at /10.56.52.132:49990"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.134Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"ou=tokens\" at /10.56.52.132:50014"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-29","timestamp":"2023-02-04T03:41:42.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-28","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":53,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-33","timestamp":"2023-02-04T03:41:53.586Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-32","client":{"ip":"10.56.48.137","port":41836},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-37","timestamp":"2023-02-04T03:41:53.615Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-36","client":{"ip":"10.56.48.137","port":41840},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-41","timestamp":"2023-02-04T03:41:53.628Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-40","client":{"ip":"10.56.48.137","port":41842},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-45","timestamp":"2023-02-04T03:41:53.641Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-44","client":{"ip":"10.56.48.137","port":41850},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-49","timestamp":"2023-02-04T03:41:54.140Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-48","client":{"ip":"10.56.49.136","port":52030},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-53","timestamp":"2023-02-04T03:41:54.170Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-52","client":{"ip":"10.56.49.136","port":52042},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-57","timestamp":"2023-02-04T03:41:54.185Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-56","client":{"ip":"10.56.49.136","port":52052},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-61","timestamp":"2023-02-04T03:41:54.200Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-60","client":{"ip":"10.56.49.136","port":52066},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-65","timestamp":"2023-02-04T03:41:54.338Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-64","client":{"ip":"10.56.51.10","port":52234},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-69","timestamp":"2023-02-04T03:41:54.366Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-68","client":{"ip":"10.56.51.10","port":52244},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-73","timestamp":"2023-02-04T03:41:54.379Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-72","client":{"ip":"10.56.51.10","port":52258},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-77","timestamp":"2023-02-04T03:41:54.392Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-76","client":{"ip":"10.56.51.10","port":52274},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/alive","headers":{"accept":["*/*"],"host":["ds-idrepo-0.ds-idrepo:8080"],"user-agent":["curl/7.74.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.766Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-0) has accepted a connection from replication server RS(ds-idrepo-2) for domain \"dc=openidm,dc=forgerock,dc=io\" at /10.56.54.132:60370"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.766Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-2) for domain \"ou=am-config\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.766Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-2) for domain \"ou=tokens\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.766Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-2) for domain \"uid=monitor\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.766Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-0) has connected to replication server RS(ds-idrepo-2) for domain \"ou=identities\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:42:05.371Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-210","timestamp":"2023-02-04T03:42:12.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-209","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-360","timestamp":"2023-02-04T03:42:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-359","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":44,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-510","timestamp":"2023-02-04T03:43:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-509","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":40,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-658","timestamp":"2023-02-04T03:43:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-657","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":40,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-805","timestamp":"2023-02-04T03:44:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-804","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":38,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":4,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-918","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.735Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-920","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":5,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-921","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.737Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-923","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":6,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-time-limit","ds-cfg-default-password-policy","ds-cfg-etime-resolution","ds-cfg-allowed-task","ds-cfg-invalid-attribute-syntax-behavior","ds-cfg-max-psearches","ds-cfg-group-id","ds-cfg-check-schema","ds-cfg-add-missing-rdn-attributes","ds-cfg-max-candidate-set-size","ds-cfg-max-internal-buffer-size","ds-cfg-restricted-client","ds-cfg-single-structural-objectclass-behavior","ds-cfg-size-limit","ds-cfg-trust-transaction-ids","ds-cfg-proxy-protocol-enabled","ds-cfg-subordinate-base-dn","ds-cfg-denied-client","ds-cfg-bind-with-dn-requires-password","ds-cfg-advertised-listen-address","ds-cfg-allowed-client","ds-cfg-server-id","ds-cfg-max-allowed-client-connections","ds-cfg-listen-address","ds-cfg-writability-mode","ds-cfg-save-config-on-successful-startup","ds-cfg-proxy-protocol-allowed-client","ds-cfg-disabled-privilege","ds-cfg-return-bind-error-messages","ds-cfg-je-backend-shared-cache-enabled","ds-cfg-unauthenticated-requests-policy","ds-cfg-idle-time-limit","ds-cfg-proxied-authorization-identity-mapper","ds-cfg-restricted-client-connection-limit","ds-cfg-group-id-failover-order","ds-cfg-notify-abandoned-operations","ds-cfg-allow-attribute-name-exceptions"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-924","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.742Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-926","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":8,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-928","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.762Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-930","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":9,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-931","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.764Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-933","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":10,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-java-class","ds-cfg-min-password-age","ds-cfg-require-change-by-time","ds-cfg-password-validator","ds-cfg-password-generator","ds-cfg-password-history-count","ds-cfg-default-password-storage-scheme","ds-cfg-allow-expired-password-changes","ds-cfg-lockout-duration","ds-cfg-allow-pre-encoded-passwords","ds-cfg-password-change-requires-current-password","ds-cfg-allow-multiple-password-values","ds-cfg-max-password-reset-age","ds-cfg-skip-validation-for-administrators","ds-cfg-last-login-time-attribute","ds-cfg-require-secure-password-changes","ds-cfg-require-secure-authentication","ds-cfg-state-update-failure-policy","ds-cfg-grace-login-count","ds-cfg-lockout-failure-expiration-interval","ds-cfg-password-expiration-warning-interval","ds-cfg-allow-user-password-changes","ds-cfg-force-change-on-reset","ds-cfg-lockout-failure-count","ds-cfg-password-history-duration","ds-cfg-expire-passwords-without-warning","ds-cfg-max-password-age","ds-cfg-last-login-time-format","ds-cfg-previous-last-login-time-format","ds-cfg-deprecated-password-storage-scheme","ds-cfg-account-status-notification-handler","ds-cfg-idle-lockout-interval","ds-cfg-password-attribute","ds-cfg-force-change-on-add"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-934","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.767Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-936","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":12,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-938","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.773Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-940","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":13,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-941","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.775Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-943","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":39,"msgId":14,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-java-class","ds-cfg-match-base-dn","ds-cfg-match-attribute","ds-cfg-enabled"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-944","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.777Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-946","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":38834},"server":{"ip":"10.56.52.4","port":4444},"request":{"protocol":"LDAPS","operation":"MODIFY","connId":39,"msgId":15,"dn":"cn=config"},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-947","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"userId":"uid=admin","timestamp":"2023-02-04T03:44:36.816Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-949","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.4","port":53440},"server":{"ip":"10.56.52.4","port":1389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":40,"msgId":3,"dn":"cn=Filtering Criteria,cn=Console LDAP Access Logger,cn=Loggers,cn=config","scope":"sub","filter":"(objectClass=*)","attrs":["ALL"]},"transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-968","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":7},"userId":"uid=admin","timestamp":"2023-02-04T03:44:42.714Z","_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-970","source":"audit","topic":"ldap-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-975","timestamp":"2023-02-04T03:44:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-974","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":38,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1052","timestamp":"2023-02-04T03:45:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1051","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":38,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-41747","timestamp":"2023-02-04T03:45:42.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-41676","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":44,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-99688","timestamp":"2023-02-04T03:46:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-99586","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":63,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-152612","timestamp":"2023-02-04T03:46:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-152538","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:47:05.389Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-203791","timestamp":"2023-02-04T03:47:12.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-203712","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":50,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-253371","timestamp":"2023-02-04T03:47:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-253302","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-302565","timestamp":"2023-02-04T03:48:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-302483","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-350867","timestamp":"2023-02-04T03:48:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-350795","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-397785","timestamp":"2023-02-04T03:49:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-397709","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":52,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-443520","timestamp":"2023-02-04T03:49:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-443454","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-489312","timestamp":"2023-02-04T03:50:12.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-489231","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-539416","timestamp":"2023-02-04T03:50:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-539338","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":50,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-588631","timestamp":"2023-02-04T03:51:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-588558","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-639361","timestamp":"2023-02-04T03:51:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-639281","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-688868","timestamp":"2023-02-04T03:52:12.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-688821","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-738515","timestamp":"2023-02-04T03:52:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-738457","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":42,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-785970","timestamp":"2023-02-04T03:53:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-785894","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-833769","timestamp":"2023-02-04T03:53:42.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-833703","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-881331","timestamp":"2023-02-04T03:54:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-881264","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":41,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-927683","timestamp":"2023-02-04T03:54:42.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-927612","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-973542","timestamp":"2023-02-04T03:55:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-973466","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002561","timestamp":"2023-02-04T03:55:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002560","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":40,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002634","timestamp":"2023-02-04T03:56:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002633","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002707","timestamp":"2023-02-04T03:56:42.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002706","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:57:06.260Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002780","timestamp":"2023-02-04T03:57:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002779","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002854","timestamp":"2023-02-04T03:57:42.824Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-1002853","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":38,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"7aaef008-9b71-4b67-93b7-23627bcb64e7-2940353","timestamp":"2023-02-04T03:58:12.823Z","eventName":"DJ-HTTP","transactionId":"7aaef008-9b71-4b67-93b7-23627bcb64e7-2934625","client":{"ip":"10.56.49.4","port":53464},"server":{"ip":"10.56.52.4","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.4:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":67,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} --- stderr --- - Looking for string "heart" - Looking for string "publish" ---------------- Pod ds-idrepo-1 ---------------- [loop_until]: kubectl --namespace=xlou logs ds-idrepo-1 [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Initializing "data/changelogDb" from Docker image Initializing "data/import-tmp" from Docker image Upgrading configuration and data... * OpenDJ data has already been upgraded to version 7.3.0.35060cde0333c76dfba29a4e88896b885a0deba7 Rebuilding degraded indexes for base DN "ou=tokens"... Rebuilding degraded indexes for base DN "ou=identities"... Rebuilding degraded indexes for base DN "ou=am-config"... Rebuilding degraded indexes for base DN "dc=openidm,dc=forgerock,dc=io"... Server configured with: Group ID : default Server ID : ds-idrepo-1 Advertised listen address : ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local Bootstrap replication server(s) : ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8989,ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8989 {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.072Z","category":"CORE","severity":"NOTICE","msgId":134,"message":"ForgeRock Directory Services 7.3.0-20230127121641-35060cde0333c76dfba29a4e88896b885a0deba7 (build 20230127141714, revision number 35060cde0333c76dfba29a4e88896b885a0deba7) starting up"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.072Z","category":"JVM","severity":"NOTICE","msgId":21,"message":"Installation Directory: /opt/opendj"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.072Z","category":"JVM","severity":"NOTICE","msgId":23,"message":"Instance Directory: /opt/opendj"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.072Z","category":"JVM","severity":"NOTICE","msgId":17,"message":"JVM Information: 17.0.6+10-LTS by Azul Systems, Inc., 64-bit architecture, 11274289152 bytes heap size"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.072Z","category":"JVM","severity":"NOTICE","msgId":20,"message":"JVM Host: ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local default/ds-idrepo-1, running Linux 5.10.147+ amd64, unknown physical memory size, number of processors available 16"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.072Z","category":"JVM","severity":"NOTICE","msgId":19,"message":"JVM Arguments: \"-XX:+UseG1GC\", \"-XX:MaxRAMPercentage=75\", \"-XX:MaxTenuringThreshold=1\", \"-Djava.security.egd=file:/dev/urandom\", \"-Dorg.opends.server.scriptName=start-ds\""} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.673Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend amCts containing 5 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:19.873Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend idmRepo containing 42 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.274Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend cfgStore containing 115 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.274Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend amIdentityStore containing 21 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.674Z","category":"SYNC","severity":"NOTICE","msgId":204,"message":"Replication server RS(ds-idrepo-1) started listening for new connections on address 0.0.0.0 port 8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.674Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 42 entries and skipped 0 in 0 seconds (average rate 2333.3/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.874Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-0) for domain \"dc=openidm,dc=forgerock,dc=io\" at ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local/10.56.52.4:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.874Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-1) has accepted a connection from directory server DS(ds-idrepo-1) for domain \"dc=openidm,dc=forgerock,dc=io\" at /10.56.52.132:52768"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.874Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-1) for domain \"dc=openidm,dc=forgerock,dc=io\" at 10.56.52.132:8989 with generation ID 555879"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.874Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-0) for domain \"uid=monitor\" at ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local/10.56.52.4:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.874Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-1) has accepted a connection from directory server DS(ds-idrepo-1) for domain \"uid=monitor\" at /10.56.52.132:52778"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:20.874Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-1) for domain \"uid=monitor\" at 10.56.52.132:8989 with generation ID 48"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 115 entries and skipped 0 in 0 seconds (average rate 6388.9/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-0) for domain \"ou=am-config\" at ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local/10.56.52.4:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-1) has accepted a connection from directory server DS(ds-idrepo-1) for domain \"ou=am-config\" at /10.56.52.132:52794"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-1) for domain \"ou=am-config\" at 10.56.52.132:8989 with generation ID 2338230"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 21 entries and skipped 0 in 0 seconds (average rate 10500.0/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-0) for domain \"ou=identities\" at ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local/10.56.52.4:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-1) has accepted a connection from directory server DS(ds-idrepo-1) for domain \"ou=identities\" at /10.56.52.132:52814"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-1) for domain \"ou=identities\" at 10.56.52.132:8989 with generation ID 253127"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.074Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 5 entries and skipped 0 in 0 seconds (average rate 5000.0/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-0) for domain \"ou=tokens\" at ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local/10.56.52.4:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-1) has accepted a connection from directory server DS(ds-idrepo-1) for domain \"ou=tokens\" at /10.56.52.132:52828"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-1) for domain \"ou=tokens\" at 10.56.52.132:8989 with generation ID 69125"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"CORE","severity":"NOTICE","msgId":818,"message":"Registered 0 static groups, 0 dynamic groups and 0 virtual static groups. The static group cache is using 0 bytes of memory"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on Administration Connector 0.0.0.0:4444"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on LDAP 0.0.0.0:1389"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on HTTP 0.0.0.0:8080"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on HTTPS 0.0.0.0:8443"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on LDAPS 0.0.0.0:1636"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"CORE","severity":"NOTICE","msgId":135,"message":"The Directory Server has started successfully"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:21.275Z","category":"CORE","severity":"NOTICE","msgId":139,"message":"The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Directory Server has started successfully"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-3","timestamp":"2023-02-04T03:41:44.605Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-2","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":136,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.714Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-1) has accepted a connection from replication server RS(ds-idrepo-2) for domain \"dc=openidm,dc=forgerock,dc=io\" at /10.56.54.132:45768"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.914Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-2) for domain \"ou=am-config\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.914Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-2) for domain \"ou=tokens\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.914Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-2) for domain \"uid=monitor\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.914Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-1) has connected to replication server RS(ds-idrepo-2) for domain \"ou=identities\" at ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local/10.56.54.132:8989"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-7","timestamp":"2023-02-04T03:42:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-6","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":57,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-71","timestamp":"2023-02-04T03:42:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-70","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":49,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:43:06.571Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-136","timestamp":"2023-02-04T03:43:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-135","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":44,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-201","timestamp":"2023-02-04T03:43:44.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-200","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":41,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-266","timestamp":"2023-02-04T03:44:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-265","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":39,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":4,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-321","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.847Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-323","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":5,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-324","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.850Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-326","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":6,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-time-limit","ds-cfg-default-password-policy","ds-cfg-etime-resolution","ds-cfg-allowed-task","ds-cfg-invalid-attribute-syntax-behavior","ds-cfg-max-psearches","ds-cfg-group-id","ds-cfg-check-schema","ds-cfg-add-missing-rdn-attributes","ds-cfg-max-candidate-set-size","ds-cfg-max-internal-buffer-size","ds-cfg-restricted-client","ds-cfg-single-structural-objectclass-behavior","ds-cfg-size-limit","ds-cfg-trust-transaction-ids","ds-cfg-proxy-protocol-enabled","ds-cfg-subordinate-base-dn","ds-cfg-denied-client","ds-cfg-bind-with-dn-requires-password","ds-cfg-advertised-listen-address","ds-cfg-allowed-client","ds-cfg-server-id","ds-cfg-max-allowed-client-connections","ds-cfg-listen-address","ds-cfg-writability-mode","ds-cfg-save-config-on-successful-startup","ds-cfg-proxy-protocol-allowed-client","ds-cfg-disabled-privilege","ds-cfg-return-bind-error-messages","ds-cfg-je-backend-shared-cache-enabled","ds-cfg-unauthenticated-requests-policy","ds-cfg-idle-time-limit","ds-cfg-proxied-authorization-identity-mapper","ds-cfg-restricted-client-connection-limit","ds-cfg-group-id-failover-order","ds-cfg-notify-abandoned-operations","ds-cfg-allow-attribute-name-exceptions"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-327","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":2,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.857Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-329","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":8,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-331","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.881Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-333","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":9,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-334","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.883Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-336","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":10,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-java-class","ds-cfg-min-password-age","ds-cfg-require-change-by-time","ds-cfg-password-validator","ds-cfg-password-generator","ds-cfg-password-history-count","ds-cfg-default-password-storage-scheme","ds-cfg-allow-expired-password-changes","ds-cfg-lockout-duration","ds-cfg-allow-pre-encoded-passwords","ds-cfg-password-change-requires-current-password","ds-cfg-allow-multiple-password-values","ds-cfg-max-password-reset-age","ds-cfg-skip-validation-for-administrators","ds-cfg-last-login-time-attribute","ds-cfg-require-secure-password-changes","ds-cfg-require-secure-authentication","ds-cfg-state-update-failure-policy","ds-cfg-grace-login-count","ds-cfg-lockout-failure-expiration-interval","ds-cfg-password-expiration-warning-interval","ds-cfg-allow-user-password-changes","ds-cfg-force-change-on-reset","ds-cfg-lockout-failure-count","ds-cfg-password-history-duration","ds-cfg-expire-passwords-without-warning","ds-cfg-max-password-age","ds-cfg-last-login-time-format","ds-cfg-previous-last-login-time-format","ds-cfg-deprecated-password-storage-scheme","ds-cfg-account-status-notification-handler","ds-cfg-idle-lockout-interval","ds-cfg-password-attribute","ds-cfg-force-change-on-add"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-337","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.887Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-339","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":12,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-341","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.894Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-343","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":13,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-345","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.896Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-347","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":28,"msgId":14,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-java-class","ds-cfg-match-base-dn","ds-cfg-match-attribute","ds-cfg-enabled"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-348","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.899Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-350","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":60600},"server":{"ip":"10.56.52.132","port":4444},"request":{"protocol":"LDAPS","operation":"MODIFY","connId":28,"msgId":15,"dn":"cn=config"},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-351","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"userId":"uid=admin","timestamp":"2023-02-04T03:44:38.937Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-353","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.52.132","port":48124},"server":{"ip":"10.56.52.132","port":1389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":29,"msgId":3,"dn":"cn=Filtering Criteria,cn=Console LDAP Access Logger,cn=Loggers,cn=config","scope":"sub","filter":"(objectClass=*)","attrs":["ALL"]},"transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-366","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":7},"userId":"uid=admin","timestamp":"2023-02-04T03:44:44.358Z","_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-368","source":"audit","topic":"ldap-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-373","timestamp":"2023-02-04T03:44:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-372","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":40,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-438","timestamp":"2023-02-04T03:45:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-437","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":50,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-503","timestamp":"2023-02-04T03:45:44.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-502","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":50,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-568","timestamp":"2023-02-04T03:46:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-567","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-633","timestamp":"2023-02-04T03:46:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-632","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":49,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-698","timestamp":"2023-02-04T03:47:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-697","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-763","timestamp":"2023-02-04T03:47:44.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-762","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":51,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:48:06.612Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-828","timestamp":"2023-02-04T03:48:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-827","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":53,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-893","timestamp":"2023-02-04T03:48:44.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-892","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-958","timestamp":"2023-02-04T03:49:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-957","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1023","timestamp":"2023-02-04T03:49:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1022","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1088","timestamp":"2023-02-04T03:50:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1087","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1153","timestamp":"2023-02-04T03:50:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1152","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1218","timestamp":"2023-02-04T03:51:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1217","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1283","timestamp":"2023-02-04T03:51:44.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1282","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":44,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1348","timestamp":"2023-02-04T03:52:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1347","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1413","timestamp":"2023-02-04T03:52:44.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1412","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":55,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1478","timestamp":"2023-02-04T03:53:14.568Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1477","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1543","timestamp":"2023-02-04T03:53:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1542","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":42,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1608","timestamp":"2023-02-04T03:54:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1607","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":52,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1673","timestamp":"2023-02-04T03:54:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1672","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1738","timestamp":"2023-02-04T03:55:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1737","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1803","timestamp":"2023-02-04T03:55:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1802","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":35,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1868","timestamp":"2023-02-04T03:56:14.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1867","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1933","timestamp":"2023-02-04T03:56:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1932","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":44,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1998","timestamp":"2023-02-04T03:57:14.567Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-1997","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":35,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"ea745f83-fb87-4608-b20d-e71c798ca6c0-2063","timestamp":"2023-02-04T03:57:44.566Z","eventName":"DJ-HTTP","transactionId":"ea745f83-fb87-4608-b20d-e71c798ca6c0-2062","client":{"ip":"10.56.49.4","port":42992},"server":{"ip":"10.56.52.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.52.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":51,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:58:06.569Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} --- stderr --- - Looking for string "heart" - Looking for string "publish" ---------------- Pod ds-idrepo-2 ---------------- [loop_until]: kubectl --namespace=xlou logs ds-idrepo-2 [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Initializing "data/changelogDb" from Docker image Initializing "data/import-tmp" from Docker image Upgrading configuration and data... * OpenDJ data has already been upgraded to version 7.3.0.35060cde0333c76dfba29a4e88896b885a0deba7 Rebuilding degraded indexes for base DN "ou=tokens"... Rebuilding degraded indexes for base DN "ou=identities"... Rebuilding degraded indexes for base DN "ou=am-config"... Rebuilding degraded indexes for base DN "dc=openidm,dc=forgerock,dc=io"... Server configured with: Group ID : default Server ID : ds-idrepo-2 Advertised listen address : ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local Bootstrap replication server(s) : ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local:8989,ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local:8989 {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.031Z","category":"CORE","severity":"NOTICE","msgId":134,"message":"ForgeRock Directory Services 7.3.0-20230127121641-35060cde0333c76dfba29a4e88896b885a0deba7 (build 20230127141714, revision number 35060cde0333c76dfba29a4e88896b885a0deba7) starting up"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.031Z","category":"JVM","severity":"NOTICE","msgId":21,"message":"Installation Directory: /opt/opendj"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.031Z","category":"JVM","severity":"NOTICE","msgId":23,"message":"Instance Directory: /opt/opendj"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.031Z","category":"JVM","severity":"NOTICE","msgId":17,"message":"JVM Information: 17.0.6+10-LTS by Azul Systems, Inc., 64-bit architecture, 11274289152 bytes heap size"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.031Z","category":"JVM","severity":"NOTICE","msgId":20,"message":"JVM Host: ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local default/ds-idrepo-2, running Linux 5.10.147+ amd64, unknown physical memory size, number of processors available 16"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.031Z","category":"JVM","severity":"NOTICE","msgId":19,"message":"JVM Arguments: \"-XX:+UseG1GC\", \"-XX:MaxRAMPercentage=75\", \"-XX:MaxTenuringThreshold=1\", \"-Djava.security.egd=file:/dev/urandom\", \"-Dorg.opends.server.scriptName=start-ds\""} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.632Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend amCts containing 5 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:57.837Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend idmRepo containing 42 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.037Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend cfgStore containing 115 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.238Z","category":"BACKEND","severity":"NOTICE","msgId":513,"message":"The database backend amIdentityStore containing 21 entries has started"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.438Z","category":"SYNC","severity":"NOTICE","msgId":204,"message":"Replication server RS(ds-idrepo-2) started listening for new connections on address 0.0.0.0 port 8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.438Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 42 entries and skipped 0 in 0 seconds (average rate 2333.3/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.638Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-0) for domain \"dc=openidm,dc=forgerock,dc=io\" at ds-idrepo-0.ds-idrepo.xlou.svc.cluster.local/10.56.52.4:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.638Z","category":"SYNC","severity":"INFORMATION","msgId":205,"message":"Replication server RS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-1) for domain \"dc=openidm,dc=forgerock,dc=io\" at ds-idrepo-1.ds-idrepo.xlou.svc.cluster.local/10.56.52.132:8989"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.638Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from directory server DS(ds-idrepo-2) for domain \"dc=openidm,dc=forgerock,dc=io\" at /10.56.54.132:42124"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.638Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-2) for domain \"dc=openidm,dc=forgerock,dc=io\" at 10.56.54.132:8989 with generation ID 555879"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-0) for domain \"ou=am-config\" at /10.56.52.4:40162"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-0) for domain \"ou=tokens\" at /10.56.52.4:40178"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-0) for domain \"uid=monitor\" at /10.56.52.4:40188"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-0) for domain \"ou=identities\" at /10.56.52.4:40192"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"ou=am-config\" at /10.56.52.132:32994"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"ou=tokens\" at /10.56.52.132:32996"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"uid=monitor\" at /10.56.52.132:32998"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:58.838Z","category":"SYNC","severity":"INFORMATION","msgId":206,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from replication server RS(ds-idrepo-1) for domain \"ou=identities\" at /10.56.52.132:33010"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from directory server DS(ds-idrepo-2) for domain \"uid=monitor\" at /10.56.54.132:42148"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-2) for domain \"uid=monitor\" at 10.56.54.132:8989 with generation ID 48"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 115 entries and skipped 0 in 0 seconds (average rate 7187.5/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from directory server DS(ds-idrepo-2) for domain \"ou=am-config\" at /10.56.54.132:42164"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-2) for domain \"ou=am-config\" at 10.56.54.132:8989 with generation ID 2338230"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 21 entries and skipped 0 in 0 seconds (average rate 10500.0/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from directory server DS(ds-idrepo-2) for domain \"ou=identities\" at /10.56.54.132:42186"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-2) for domain \"ou=identities\" at 10.56.54.132:8989 with generation ID 253127"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.038Z","category":"BACKEND","severity":"NOTICE","msgId":515,"message":"Exported 5 entries and skipped 0 in 0 seconds (average rate 5000.0/sec)"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"SYNC","severity":"INFORMATION","msgId":207,"message":"Replication server RS(ds-idrepo-2) has accepted a connection from directory server DS(ds-idrepo-2) for domain \"ou=tokens\" at /10.56.54.132:42196"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"SYNC","severity":"NOTICE","msgId":62,"message":"Directory server DS(ds-idrepo-2) has connected to replication server RS(ds-idrepo-2) for domain \"ou=tokens\" at 10.56.54.132:8989 with generation ID 69125"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"CORE","severity":"NOTICE","msgId":818,"message":"Registered 0 static groups, 0 dynamic groups and 0 virtual static groups. The static group cache is using 0 bytes of memory"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on Administration Connector 0.0.0.0:4444"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on LDAP 0.0.0.0:1389"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on HTTP 0.0.0.0:8080"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on HTTPS 0.0.0.0:8443"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"PROTOCOL","severity":"NOTICE","msgId":276,"message":"Started listening for new connections on LDAPS 0.0.0.0:1636"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"CORE","severity":"NOTICE","msgId":135,"message":"The Directory Server has started successfully"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:41:59.239Z","category":"CORE","severity":"NOTICE","msgId":139,"message":"The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Directory Server has started successfully"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-119","timestamp":"2023-02-04T03:42:20.999Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-118","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":128,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-367","timestamp":"2023-02-04T03:42:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-366","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":52,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:43:06.504Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-464","timestamp":"2023-02-04T03:43:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-463","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-559","timestamp":"2023-02-04T03:43:50.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-558","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-654","timestamp":"2023-02-04T03:44:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-653","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":39,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":4,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-722","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.007Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-724","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":5,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-725","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.009Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-727","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":6,"dn":"cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-time-limit","ds-cfg-default-password-policy","ds-cfg-etime-resolution","ds-cfg-allowed-task","ds-cfg-invalid-attribute-syntax-behavior","ds-cfg-max-psearches","ds-cfg-group-id","ds-cfg-check-schema","ds-cfg-add-missing-rdn-attributes","ds-cfg-max-candidate-set-size","ds-cfg-max-internal-buffer-size","ds-cfg-restricted-client","ds-cfg-single-structural-objectclass-behavior","ds-cfg-size-limit","ds-cfg-trust-transaction-ids","ds-cfg-proxy-protocol-enabled","ds-cfg-subordinate-base-dn","ds-cfg-denied-client","ds-cfg-bind-with-dn-requires-password","ds-cfg-advertised-listen-address","ds-cfg-allowed-client","ds-cfg-server-id","ds-cfg-max-allowed-client-connections","ds-cfg-listen-address","ds-cfg-writability-mode","ds-cfg-save-config-on-successful-startup","ds-cfg-proxy-protocol-allowed-client","ds-cfg-disabled-privilege","ds-cfg-return-bind-error-messages","ds-cfg-je-backend-shared-cache-enabled","ds-cfg-unauthenticated-requests-policy","ds-cfg-idle-time-limit","ds-cfg-proxied-authorization-identity-mapper","ds-cfg-restricted-client-connection-limit","ds-cfg-group-id-failover-order","ds-cfg-notify-abandoned-operations","ds-cfg-allow-attribute-name-exceptions"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-728","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.013Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-730","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":8,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-732","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.033Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-734","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":9,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-735","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.035Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-737","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":10,"dn":"cn=Default Password Policy,cn=Password Policies,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-java-class","ds-cfg-min-password-age","ds-cfg-require-change-by-time","ds-cfg-password-validator","ds-cfg-password-generator","ds-cfg-password-history-count","ds-cfg-default-password-storage-scheme","ds-cfg-allow-expired-password-changes","ds-cfg-lockout-duration","ds-cfg-allow-pre-encoded-passwords","ds-cfg-password-change-requires-current-password","ds-cfg-allow-multiple-password-values","ds-cfg-max-password-reset-age","ds-cfg-skip-validation-for-administrators","ds-cfg-last-login-time-attribute","ds-cfg-require-secure-password-changes","ds-cfg-require-secure-authentication","ds-cfg-state-update-failure-policy","ds-cfg-grace-login-count","ds-cfg-lockout-failure-expiration-interval","ds-cfg-password-expiration-warning-interval","ds-cfg-allow-user-password-changes","ds-cfg-force-change-on-reset","ds-cfg-lockout-failure-count","ds-cfg-password-history-duration","ds-cfg-expire-passwords-without-warning","ds-cfg-max-password-age","ds-cfg-last-login-time-format","ds-cfg-previous-last-login-time-format","ds-cfg-deprecated-password-storage-scheme","ds-cfg-account-status-notification-handler","ds-cfg-idle-lockout-interval","ds-cfg-password-attribute","ds-cfg-force-change-on-add"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-738","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.038Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-740","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":12,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["1.1"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-742","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.046Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-744","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":13,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["objectclass"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-745","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":0,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.048Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-747","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"SEARCH","connId":55,"msgId":14,"dn":"cn=Exact Match,cn=Identity Mappers,cn=config","scope":"base","filter":"(objectClass=*)","attrs":["ds-cfg-java-class","ds-cfg-match-base-dn","ds-cfg-match-attribute","ds-cfg-enabled"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-748","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":1},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.051Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-750","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":33308},"server":{"ip":"10.56.54.132","port":4444},"request":{"protocol":"LDAPS","operation":"MODIFY","connId":55,"msgId":15,"dn":"cn=config"},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-751","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":35,"elapsedTimeUnits":"MILLISECONDS"},"userId":"uid=admin","timestamp":"2023-02-04T03:44:41.091Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-753","source":"audit","topic":"ldap-access","level":"INFO"} {"eventName":"DJ-LDAP","client":{"ip":"10.56.54.132","port":51026},"server":{"ip":"10.56.54.132","port":1389},"request":{"protocol":"LDAP","operation":"SEARCH","connId":56,"msgId":3,"dn":"cn=Filtering Criteria,cn=Console LDAP Access Logger,cn=Loggers,cn=config","scope":"sub","filter":"(objectClass=*)","attrs":["ALL"]},"transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-763","response":{"status":"SUCCESSFUL","statusCode":"0","elapsedTime":1,"elapsedTimeUnits":"MILLISECONDS","nentries":7},"userId":"uid=admin","timestamp":"2023-02-04T03:44:45.977Z","_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-765","source":"audit","topic":"ldap-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-794","timestamp":"2023-02-04T03:44:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-793","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":41,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-891","timestamp":"2023-02-04T03:45:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-890","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":42,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-990","timestamp":"2023-02-04T03:45:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-989","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":54,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1092","timestamp":"2023-02-04T03:46:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1091","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":49,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1193","timestamp":"2023-02-04T03:46:50.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1192","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":51,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1296","timestamp":"2023-02-04T03:47:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1295","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":50,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1398","timestamp":"2023-02-04T03:47:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1397","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":54,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:48:06.356Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1501","timestamp":"2023-02-04T03:48:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1500","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":50,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1603","timestamp":"2023-02-04T03:48:50.966Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1602","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1706","timestamp":"2023-02-04T03:49:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1705","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1808","timestamp":"2023-02-04T03:49:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1807","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-1911","timestamp":"2023-02-04T03:50:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-1910","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":40,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2013","timestamp":"2023-02-04T03:50:50.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2012","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":44,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2116","timestamp":"2023-02-04T03:51:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2115","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2218","timestamp":"2023-02-04T03:51:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2217","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":48,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2320","timestamp":"2023-02-04T03:52:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2319","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":46,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2424","timestamp":"2023-02-04T03:52:50.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2423","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":47,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2528","timestamp":"2023-02-04T03:53:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2527","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":45,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2632","timestamp":"2023-02-04T03:53:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2631","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":43,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2741","timestamp":"2023-02-04T03:54:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2740","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":41,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2850","timestamp":"2023-02-04T03:54:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2849","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":41,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-2960","timestamp":"2023-02-04T03:55:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-2959","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":42,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-3070","timestamp":"2023-02-04T03:55:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-3069","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":39,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-3180","timestamp":"2023-02-04T03:56:20.964Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-3179","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-3290","timestamp":"2023-02-04T03:56:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-3289","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-3400","timestamp":"2023-02-04T03:57:20.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-3399","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":34,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"_id":"c02e258a-fb94-44ad-99c6-18704c1778f0-3514","timestamp":"2023-02-04T03:57:50.963Z","eventName":"DJ-HTTP","transactionId":"c02e258a-fb94-44ad-99c6-18704c1778f0-3513","client":{"ip":"10.56.49.4","port":35522},"server":{"ip":"10.56.54.132","port":8080},"http":{"request":{"secure":false,"method":"GET","path":"http://ds-idrepo-2.ds-idrepo.xlou.svc.cluster.local:8080/metrics/prometheus","headers":{"accept":["application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1"],"host":["10.56.54.132:8080"],"user-agent":["Prometheus/2.41.0"]}}},"response":{"status":"SUCCESSFUL","statusCode":"200","elapsedTime":35,"elapsedTimeUnits":"MILLISECONDS"},"source":"audit","topic":"http-access","level":"INFO"} {"eventName":"DJ-SERVER","timestamp":"2023-02-04T03:58:06.320Z","category":"SYNC","severity":"WARNING","msgId":383,"message":"The change number indexer cannot advance because replica 'ds-idrepo-0' has not sent any messages for a while. Verify whether the replica is still running or is connected to a replication server"} --- stderr --- - Looking for string "heart" - Looking for string "publish" ________________________________________________________________________________ [04/Feb/2023 03:58:15] CheckReplication_ds-idrepo step5 : Compare ds-sync-state for suffixes across all replicas ________________________________________________________________________________ [run_command]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b " " -s base "objectclass=*" namingContexts [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: namingContexts: uid=admin namingContexts: ou=tokens namingContexts: uid=monitor namingContexts: dc=openidm,dc=forgerock,dc=io namingContexts: ou=am-config namingContexts: ou=identities namingContexts: uid=proxy --- stderr --- => Using namingContexts: {'uid=proxy', 'ou=tokens', 'ou=identities'} [run_command]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "uid=proxy" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: uid=proxy --- stderr --- => Skipping suffix "uid=proxy" as "ds-sync-state" not found in stdout [run_command]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "ou=tokens" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: ou=tokens ds-sync-state: 010b01861a831b7200000011ds-idrepo-0 --- stderr --- [run_command]: kubectl --namespace=xlou exec ds-idrepo-1 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "ou=tokens" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: ou=tokens ds-sync-state: 010b01861a831b7200000011ds-idrepo-0 --- stderr --- [run_command]: kubectl --namespace=xlou exec ds-idrepo-2 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "ou=tokens" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: ou=tokens ds-sync-state: 010b01861a831b7200000011ds-idrepo-0 --- stderr --- [run_command]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "ou=identities" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: ou=identities ds-sync-state: 010b01861a92b396000f5548ds-idrepo-0 --- stderr --- [run_command]: kubectl --namespace=xlou exec ds-idrepo-1 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "ou=identities" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: ou=identities ds-sync-state: 010b01861a92b396000f5548ds-idrepo-0 --- stderr --- [run_command]: kubectl --namespace=xlou exec ds-idrepo-2 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "ou=identities" -s base "objectclass=*" ds-sync-state [run_command]: OK (rc = 0 - expected to be in [0]) --- stdout --- dn: ou=identities ds-sync-state: 010b01861a92b396000f5548ds-idrepo-0 --- stderr --- => loop 1 out of 30 : repl_state = {'ou=tokens': {'ds-idrepo-0': ['010b01861a831b7200000011ds-idrepo-0'], 'ds-idrepo-1': ['010b01861a831b7200000011ds-idrepo-0'], 'ds-idrepo-2': ['010b01861a831b7200000011ds-idrepo-0']}, 'ou=identities': {'ds-idrepo-0': ['010b01861a92b396000f5548ds-idrepo-0'], 'ds-idrepo-1': ['010b01861a92b396000f5548ds-idrepo-0'], 'ds-idrepo-2': ['010b01861a92b396000f5548ds-idrepo-0']}} => ds-sync-state match for "ou=tokens" => ds-sync-state match for "ou=identities" => {'uid=proxy', 'ou=tokens', 'ou=identities'} expunged. set() remain SUCCESS: replication seems to be synchronised ________________________________________________________________________________ [04/Feb/2023 03:58:31] CheckReplication_ds-idrepo step6 : Check replication is in sync by adding once and searching all ________________________________________________________________________________ [loop_until]: kubectl --namespace=xlou cp /tmp/tmp4m_465mq -c ds ds-idrepo-0:/tmp/entries.ldif [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapmodify --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -f /tmp/entries.ldif [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- # ADD operation successful for DN uid=pyrock_tvehp,ou=identities --- stderr --- => Added test entry with dn "uid=pyrock_tvehp,ou=identities" => loop 1 out of 5 to check "3" pods for presence of test entry "uid=pyrock_tvehp,ou=identities" [run_command]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "uid=pyrock_tvehp,ou=identities" -s base objectclass=* 1.1 [run_command]: OK (rc = 0 - expected to be in [0, 32]) --- stdout --- dn: uid=pyrock_tvehp,ou=identities --- stderr --- => Found test entry "uid=pyrock_tvehp,ou=identities" on "ds-idrepo-0" [run_command]: kubectl --namespace=xlou exec ds-idrepo-1 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "uid=pyrock_tvehp,ou=identities" -s base objectclass=* 1.1 [run_command]: OK (rc = 0 - expected to be in [0, 32]) --- stdout --- dn: uid=pyrock_tvehp,ou=identities --- stderr --- => Found test entry "uid=pyrock_tvehp,ou=identities" on "ds-idrepo-1" [run_command]: kubectl --namespace=xlou exec ds-idrepo-2 -c ds -- ldapsearch --noPropertiesFile -p 1389 --useStartTls --trustAll -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" -b "uid=pyrock_tvehp,ou=identities" -s base objectclass=* 1.1 [run_command]: OK (rc = 0 - expected to be in [0, 32]) --- stdout --- dn: uid=pyrock_tvehp,ou=identities --- stderr --- => Found test entry "uid=pyrock_tvehp,ou=identities" on "ds-idrepo-2" [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 -c ds -- ldapdelete -p 4444 -Z -X -D "uid=admin" -w "k7raQF3XHmafcPfKXqlq3PYJhqhbkC3C" uid=pyrock_tvehp,ou=identities [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- # DELETE operation successful for DN uid=pyrock_tvehp,ou=identities --- stderr --- => Deleted test entry with dn "uid=pyrock_tvehp,ou=identities" SUCCESS: replication seems to be synchronised ________________________________________________________________________________ [04/Feb/2023 03:58:52] CheckReplication_ds-idrepo post : N/A ________________________________________________________________________________ SUCCESS : replication seems to be synchronised Setting result to PASS Task has been successfully stopped