--Task-- name: revert_environment_variables enabled: True class_name: RevertEnvironmentVariables source_name: controller source_namespace: >default< target_name: controller target_namespace: >default< start: 0.0 stop: None timeout: not yet initialized for this Task as timeout was not explicitly defined, see updates below loop: False interval: None dependencies: [] wait_for: [] preceding_task: None options: {} group_name: None Current dir: /mnt/disk1/xslou/workshop/lodestar-fork/pyrock ________________________________________________________________________________ [2024-11-13 11:43:18] revert_environment_variables pre : N/A ________________________________________________________________________________ 2024-11-13 11:43:18 - INFO: Timeout for this Task has changed to 12h (43200 seconds) 2024-11-13 11:43:18 - INFO: timeout was set to Task default (2x global_duration or 300 seconds (whatever is higher)), because it was unset ________________________________________________________________________________ [2024-11-13 11:43:18] revert_environment_variables step1 : N/A ________________________________________________________________________________ 2024-11-13 11:43:18,700 INFO Reverting environment variables for am 2024-11-13 11:43:19,098 ERROR [loop_until]: Function failed after 0s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:24,451 ERROR [loop_until]: Function failed after 5s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:29,808 ERROR [loop_until]: Function failed after 11s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:35,164 ERROR [loop_until]: Function failed after 16s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:40,520 ERROR [loop_until]: Function failed after 21s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:45,873 ERROR [loop_until]: Function failed after 27s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:51,226 ERROR [loop_until]: Function failed after 32s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:43:56,579 ERROR [loop_until]: Function failed after 37s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:01,933 ERROR [loop_until]: Function failed after 43s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:07,284 ERROR [loop_until]: Function failed after 48s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:12,640 ERROR [loop_until]: Function failed after 53s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:17,996 ERROR [loop_until]: Function failed after 59s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:23,350 ERROR [loop_until]: Function failed after 1m 04s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:28,702 ERROR [loop_until]: Function failed after 1m 10s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:34,052 ERROR [loop_until]: Function failed after 1m 15s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:39,405 ERROR [loop_until]: Function failed after 1m 20s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:44,758 ERROR [loop_until]: Function failed after 1m 26s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:50,114 ERROR [loop_until]: Function failed after 1m 31s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:44:55,465 ERROR [loop_until]: Function failed after 1m 36s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:00,816 ERROR [loop_until]: Function failed after 1m 42s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:06,169 ERROR [loop_until]: Function failed after 1m 47s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:11,568 ERROR [loop_until]: Function failed after 1m 52s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:16,917 ERROR [loop_until]: Function failed after 1m 58s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:22,272 ERROR [loop_until]: Function failed after 2m 03s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:27,623 ERROR [loop_until]: Function failed after 2m 08s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:32,976 ERROR [loop_until]: Function failed after 2m 14s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:38,333 ERROR [loop_until]: Function failed after 2m 19s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:43,686 ERROR [loop_until]: Function failed after 2m 24s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:49,036 ERROR [loop_until]: Function failed after 2m 30s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:54,395 ERROR [loop_until]: Function failed after 2m 35s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:45:59,750 ERROR [loop_until]: Function failed after 2m 41s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:46:05,110 ERROR [loop_until]: Function failed after 2m 46s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:46:10,472 ERROR [loop_until]: Function failed after 2m 51s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:46:15,826 ERROR [loop_until]: Function failed after 2m 57s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:46:21,180 ERROR [loop_until]: Function failed after 3m 02s (rc=1 - expected to be in [0]) - retry; CMD output was: CMD error was: Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:46:21,180 ERROR [loop_until]: rc=1, expected [0] 2024-11-13 11:46:21,180 DEBUG --- stdout --- 2024-11-13 11:46:21,181 DEBUG 2024-11-13 11:46:21,181 DEBUG --- stderr --- 2024-11-13 11:46:21,181 DEBUG Error from server (NotFound): namespaces "fr-platform" not found 2024-11-13 11:46:21 - INFO: Traceback (most recent call last): File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 603, in run self._run() File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/StepTask.py", line 54, in _run self._run_method(step) File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/lib/scheduler/tasks/Task.py", line 644, in _run_method exec("self.%s()" % method) File "", line 1, in File "/mnt/disk1/xslou/workshop/lodestar-fork/pyrock/tasks/deployment/utility_actions.py", line 47, in step1 pyrock_run.deployment.revert_env_vars() File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/multinamespace_deployment.py", line 146, in revert_env_vars deployment.revert_env_vars() File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/deployment_idc.py", line 157, in revert_env_vars component.revert_env_vars() File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/components/component.py", line 1162, in revert_env_vars component = kubectl(f'get {self.k8s_type} {self.instance_name} --output json', component=self, verbose=False).stdout ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/cloud_utils.py", line 108, in kubectl return __kubectl(args, namespace, no_loop=no_loop, output_file=output_file, expected_rc=expected_rc, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/cloud_utils.py", line 182, in __kubectl return cmd.loop_until(command, max_time=max_time, interval=interval, verbose=verbose, expected_rc=expected_rc, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/disk1/xslou/workshop/lodestar-fork/shared/lib/utils/cmd/cmd.py", line 578, in loop_until raise FailException(message=exception_message) shared.lib.utils.exception.FailException: [loop_until]: rc=1, expected [0] Setting result to FAIL Task has been successfully stopped