--Task-- name: DisasterRecovery_ds-idrepo enabled: True class_name: DSDisasterRecoveryTask source_name: overseer-0 source_namespace: >default< target_name: ds-idrepo target_namespace: >default< start: 0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates bellow loop: False interval: None dependencies: [] wait_for: [] options: {} group_name: None Current dir: /mnt/disks/data/xslou/lodestar-fork/pyrock ________________________________________________________________________________ [13/Jun/2023 15:32:45] DisasterRecovery_ds-idrepo pre : Prepare the task ________________________________________________________________________________ [13/Jun/2023 15:32:45] - INFO: Timeout for this Task has changed to 12h (43200 seconds) [13/Jun/2023 15:32:45] - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [13/Jun/2023 15:32:45] DisasterRecovery_ds-idrepo step1 : Start disaster recovery process ________________________________________________________________________________ [13/Jun/2023 15:32:45] - INFO: Start disaster recovery [loop_until]: kubectl --namespace=xlou cp /mnt/disks/data/xslou/lodestar-fork/ext/keys/ds-backup.json ds-idrepo-0:/opt/opendj/ds-backup.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Defaulted container "ds" out of: ds, initialize (init) [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 --container ds -- dsbackup list --last --noPropertiesFile --backupLocation gs://performance-bucket-data-eu/perf-tests/7.4-10M-users/ds-idrepo-0 --storageProperty gs.credentials.path:/opt/opendj/ds-backup.json [loop_until]: (max_time=600, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Backend name: idmRepo Server ID: ds-idrepo-0 Backup Date: 03/Apr/2023 20:51:32 [Etc/UTC] Backup ID: idmRepo_20230403205132073 Backend name: amIdentityStore Server ID: ds-idrepo-0 Backup Date: 03/Apr/2023 21:25:03 [Etc/UTC] Backup ID: amIdentityStore_20230403212503980 Found 2 backup(s) with the requested characteristics --- stderr --- Found ['idmRepo', 'amIdentityStore'] backends in the storage bucket. start disaster recovery of dc=openidm,dc=forgerock,dc=io. [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 --container ds -- /opt/opendj/bin/dsrepl start-disaster-recovery --trustAll --bindPassword 1S2yr2Gp9T0ZiKG7kZHUWb10d0YJyJg6 --baseDN "dc=openidm,dc=forgerock,dc=io" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Starting disaster recovery for server 'ds-idrepo-0' for base DNs: 'dc=openidm,dc=forgerock,dc=io' Starting disaster recovery for base DN: 'dc=openidm,dc=forgerock,dc=io' Done Replication is now suspended and the replicated servers will reject any modifications. Now initialize the contents of the base DNs on all of the servers by importing LDIF or by restoring a backup. The same LDIF or backup must be used on each server. When the initialization has completed you must use the 'end-disaster-recovery' subcommand to resume replication for the base DNs. --- stderr --- start disaster recovery of ou=identities. [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 --container ds -- /opt/opendj/bin/dsrepl start-disaster-recovery --trustAll --bindPassword 1S2yr2Gp9T0ZiKG7kZHUWb10d0YJyJg6 --baseDN "ou=identities" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Starting disaster recovery for server 'ds-idrepo-0' for base DNs: 'ou=identities' Starting disaster recovery for base DN: 'ou=identities' Done Replication is now suspended and the replicated servers will reject any modifications. Now initialize the contents of the base DNs on all of the servers by importing LDIF or by restoring a backup. The same LDIF or backup must be used on each server. When the initialization has completed you must use the 'end-disaster-recovery' subcommand to resume replication for the base DNs. --- stderr --- ________________________________________________________________________________ [13/Jun/2023 15:32:58] DisasterRecovery_ds-idrepo step2 : End disaster recovery process ________________________________________________________________________________ [13/Jun/2023 15:32:58] - INFO: Ending disaster recovery [loop_until]: kubectl --namespace=xlou cp /mnt/disks/data/xslou/lodestar-fork/ext/keys/ds-backup.json ds-idrepo-0:/opt/opendj/ds-backup.json [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- --- stderr --- Defaulted container "ds" out of: ds, initialize (init) [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 --container ds -- dsbackup list --last --noPropertiesFile --backupLocation gs://performance-bucket-data-eu/perf-tests/7.4-10M-users/ds-idrepo-0 --storageProperty gs.credentials.path:/opt/opendj/ds-backup.json [loop_until]: (max_time=600, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Backend name: idmRepo Server ID: ds-idrepo-0 Backup Date: 03/Apr/2023 20:51:32 [Etc/UTC] Backup ID: idmRepo_20230403205132073 Backend name: amIdentityStore Server ID: ds-idrepo-0 Backup Date: 03/Apr/2023 21:25:03 [Etc/UTC] Backup ID: amIdentityStore_20230403212503980 Found 2 backup(s) with the requested characteristics --- stderr --- Found ['idmRepo', 'amIdentityStore'] backends in the storage bucket. end disaster recovery of dc=openidm,dc=forgerock,dc=io. [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 --container ds -- /opt/opendj/bin/dsrepl end-disaster-recovery --trustAll --bindPassword 1S2yr2Gp9T0ZiKG7kZHUWb10d0YJyJg6 --baseDN "dc=openidm,dc=forgerock,dc=io" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Ending disaster recovery for server 'ds-idrepo-0' for base DNs: 'dc=openidm,dc=forgerock,dc=io' Ending disaster recovery for base DN 'dc=openidm,dc=forgerock,dc=io' Done Replication has now resumed for the base DNs and directory servers will accept modifications. --- stderr --- end disaster recovery of ou=identities. [loop_until]: kubectl --namespace=xlou exec ds-idrepo-0 --container ds -- /opt/opendj/bin/dsrepl end-disaster-recovery --trustAll --bindPassword 1S2yr2Gp9T0ZiKG7kZHUWb10d0YJyJg6 --baseDN "ou=identities" [loop_until]: (max_time=180, interval=5, expected_rc=[0] [loop_until]: OK (rc = 0) --- stdout --- Ending disaster recovery for server 'ds-idrepo-0' for base DNs: 'ou=identities' Ending disaster recovery for base DN 'ou=identities' Done Replication has now resumed for the base DNs and directory servers will accept modifications. --- stderr --- ________________________________________________________________________________ [13/Jun/2023 15:33:11] DisasterRecovery_ds-idrepo post : Post method ________________________________________________________________________________ Setting result to PASS Task has been successfully stopped