[LYN-2470] Integrate resource mapping tool with engine python environment (#17)

1. Add support to use engine python runtime environment. Ideally we should use project executable directory which contains required qt binaries after build. But currently Qt plugins are not put under proper subdirectory which causes huge overhead of loading plugin binaries (see details in this jira https://jira.agscollab.com/browse/LYN-2669 ). The temporary solution is to use AWSCore.Editor target to create a subdirectory for our use case.   
2. **Minor Fix** After migrating to engine python environment, see boto3 warning while using debug mode, seems like a existing issue for a long time https://github.com/boto/boto3/issues/454. Register a after call event to close connection   
3. **Minor Fix** After migrating to engine python environment, see Qt multithread warning while using debug mode, change direct setter function call to qt signal
This commit is contained in:
Vincent Liu
2021-04-23 11:34:37 -07:00
committed by GitHub
parent 16ca036ea8
commit a4e1abf6df
20 changed files with 343 additions and 126 deletions
+1
View File
@@ -70,6 +70,7 @@ if (PAL_TRAIT_BUILD_HOST_TOOLS)
ly_add_target(
NAME AWSCore.Editor MODULE
NAMESPACE Gem
OUTPUT_SUBDIRECTORY AWSCoreEditorPlugins
FILES_CMAKE
awscore_editor_shared_files.cmake
INCLUDE_DIRECTORIES
@@ -1,7 +1,50 @@
# Welcome to the AWS Core Resource Mapping Tool project!
This project is set up like a standard Python project. The initialization
## Setup aws config and credential
Resource mapping tool is using boto3 to interact with aws services:
* Follow boto3
[Configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html) to setup default aws region.
* Follow boto3
[Credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) to setup default profile or credential keys.
Or follow **AWS CLI** configuration which can be reused by boto3 lib:
* Follow
[Quick configuration with aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config)
**In Progress** - Override default aws profile in resource mapping tool
## Python Environment Setup Options
### 1. Engine python environment
In order to use engine python environment, it requires to link Qt binaries for this tool.
Follow cmake instructions to configure your project, for example:
```
$ cmake -B <BUILD_FOLDER> -S . -G "Visual Studio 16 2019" -DLY_3RDPARTY_PATH=<PATH_TO_3RDPARTY> -DLY_PROJECTS=<PROJECT_NAME>
```
Build project with **AWSCore.Editor** target to generate required Qt binaries.
(Or use **Editor** target)
```
$ cmake --build <BUILD_FOLDER> --target AWSCore.Editor --config <CONFIG> -j <NUM_JOBS>
```
Launch resource mapping tool under engine root folder:
#### Windows
##### release mode
```
$ python\python.cmd Gems\AWSCore\Code\Tools\ResourceMappingTool\resource_mapping_tool.py --binaries_path <PATH_TO_BUILD_FOLDER>\bin\profile\AWSCoreEditorPlugins
```
##### debug mode
```
$ python\python.cmd debug Gems\AWSCore\Code\Tools\ResourceMappingTool\resource_mapping_tool.py --binaries_path <PATH_TO_BUILD_FOLDER>\bin\debug\AWSCoreEditorPlugins
```
### 2. Python virtual environment
This project is set up like a standard Python project. The initialization
process also creates a virtualenv within this project, stored under the `.env`
directory. To create the virtualenv it assumes that there is a `python3`
(or `python` for Windows) executable in your path with access to the `venv`
@@ -32,28 +75,15 @@ Once the virtualenv is activated, you can install the required dependencies.
$ pip install -r requirements.txt
```
## Setup aws config and credential
Resource mapping tool is using boto3 to interact with aws services:
* Follow boto3
[Configuration](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html) to setup default aws region.
* Follow boto3
[Credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) to setup default profile or credential keys.
Or follow **AWS CLI** configuration which can be reused by boto3 lib:
* Follow
[Quick configuration with aws configure](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config)
**In Progress** - Override default aws profile in resource mapping tool
## Launch Options
### 1. Launch Resource Mapping Tool from python directly
#### 2.1 Launch Options
##### 2.1.1 Launch Resource Mapping Tool from python directly
At this point you can launch tool like other standard python project.
```
$ python resource_mapping_tool.py
```
### 2. Launch Resource Mapping Tool from batch script/Editor
##### 2.1.2 Launch Resource Mapping Tool from batch script/Editor
Update `resource_mapping_tool.cmd` with your virtualenv full path.
* **VIRTUALENV_PATH**: Fill this variable with your virtualenv full path.
@@ -28,13 +28,12 @@ logger = logging.getLogger(__name__)
class ImportResourcesController(QObject):
add_import_resources = Signal(list)
add_import_resources_sender: Signal = Signal(list)
set_notification_frame_text_sender: Signal = Signal(str)
"""
ImportResourcesController is the place to bind ImportResource view with its
corresponding behavior
TODO: add error handling once it is ready
"""
def __init__(self) -> None:
@@ -44,6 +43,8 @@ class ImportResourcesController(QObject):
self._view_manager: ViewManager = ViewManager.get_instance()
# Initialize view and model related references
self._import_resources_page: ImportResourcesPage = self._view_manager.get_import_resources_page()
self.set_notification_frame_text_sender.connect(
self._import_resources_page.notification_frame.set_frame_text_receiver)
self._tree_view: ResourceTreeView = self._import_resources_page.tree_view
self._proxy_model: ResourceProxyModel = self._tree_view.resource_proxy_model
@@ -60,11 +61,10 @@ class ImportResourcesController(QObject):
self._proxy_model.deduplicate_selected_import_resources(self._tree_view.selectedIndexes())
if unique_resources:
logger.debug(f"Importing selected resources: {unique_resources} ...")
self.add_import_resources.emit(unique_resources)
self.add_import_resources_sender.emit(unique_resources)
self._back_to_view_edit_page()
else:
self._import_resources_page.set_notification_frame_text(
error_messages.IMPORT_RESOURCES_PAGE_NO_RESOURCES_SELECTED_ERROR_MESSAGE)
self.set_notification_frame_text_sender.emit(error_messages.IMPORT_RESOURCES_PAGE_NO_RESOURCES_SELECTED_ERROR_MESSAGE)
def _start_search_resources_async(self) -> None:
configuration: Configuration = self._configuration_manager.configuration
@@ -77,8 +77,7 @@ class ImportResourcesController(QObject):
async_worker = FunctionWorker(self._request_cfn_resources_callback, configuration.region)
async_worker.signals.result.connect(self._load_cfn_resources_callback)
else:
self._import_resources_page.set_notification_frame_text(
error_messages.IMPORT_RESOURCES_PAGE_SEARCH_VERSION_ERROR_MESSAGE)
self.set_notification_frame_text_sender.emit(error_messages.IMPORT_RESOURCES_PAGE_SEARCH_VERSION_ERROR_MESSAGE)
return
self._tree_view.reset_view()
@@ -100,7 +99,7 @@ class ImportResourcesController(QObject):
resources[stack_name] = resource_type_and_names
return resources
except RuntimeError as e:
self._import_resources_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
def _request_typed_resources_callback(self, region: str) -> List[str]:
resource_type_index: int = self._import_resources_page.typed_resources_combobox.currentIndex()
@@ -115,12 +114,11 @@ class ImportResourcesController(QObject):
elif resource_type_index == constants.AWS_RESOURCE_S3_BUCKET_INDEX:
resources = aws_utils.list_s3_buckets(region)
else:
self._import_resources_page.set_notification_frame_text(
error_messages.IMPORT_RESOURCES_PAGE_RESOURCE_TYPE_ERROR_MESSAGE)
self.set_notification_frame_text_sender.emit(error_messages.IMPORT_RESOURCES_PAGE_RESOURCE_TYPE_ERROR_MESSAGE)
return resources
except RuntimeError as e:
self._import_resources_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
def _load_cfn_resources_callback(self, resources: Dict[str, List[BasicResourceAttributes]]) -> None:
if not resources:
@@ -179,7 +177,7 @@ class ImportResourcesController(QObject):
def reset_page(self):
"""Reset import resources page to its default state"""
self._tree_view.reset_view()
self._import_resources_page.hide_notification_frame()
self._import_resources_page.notification_frame.setVisible(False)
self._import_resources_page.set_current_main_view_index(ImportResourcesPageConstants.TREE_VIEW_PAGE_INDEX)
self._import_resources_page.typed_resources_combobox.setCurrentIndex(-1)
self._import_resources_page.search_version = None
@@ -10,7 +10,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
"""
import logging
from PySide2.QtCore import (QCoreApplication, QModelIndex, QObject, Slot)
from PySide2.QtCore import (QCoreApplication, QModelIndex, QObject, Signal, Slot)
from PySide2.QtWidgets import QFileDialog
from typing import (Dict, List)
@@ -32,6 +32,9 @@ logger = logging.getLogger(__name__)
class ViewEditController(QObject):
set_notification_frame_text_sender: Signal = Signal(str)
set_notification_page_frame_text_sender: Signal = Signal(str)
"""
ViewEditController is the place to bind ViewEdit view with its
corresponding behavior
@@ -45,6 +48,10 @@ class ViewEditController(QObject):
self._view_manager: ViewManager = ViewManager.get_instance()
# Initialize view and model related references
self._view_edit_page: ViewEditPage = self._view_manager.get_view_edit_page()
self.set_notification_frame_text_sender.connect(
self._view_edit_page.notification_frame.set_frame_text_receiver)
self.set_notification_page_frame_text_sender.connect(
self._view_edit_page.notification_page_frame.set_frame_text_receiver)
self._table_view: ResourceTableView = self._view_edit_page.table_view
self._proxy_model: ResourceProxyModel = self._table_view.resource_proxy_model
@@ -78,7 +85,7 @@ class ViewEditController(QObject):
return True
except IOError as e:
logger.exception(e)
self._view_edit_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
return False
def _convert_and_load_into_model(self, config_file_name: str) -> None:
@@ -92,7 +99,7 @@ class ViewEditController(QObject):
resources = json_utils.convert_json_dict_to_resources(self._config_file_json_source)
except (IOError, ValueError, KeyError) as e:
logger.exception(e)
self._view_edit_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
self._view_edit_page.set_table_view_page_interactions_enabled(False)
# load resources into model
@@ -109,7 +116,7 @@ class ViewEditController(QObject):
new_config_file_path, configuration.account_id, configuration.region)
except IOError as e:
logger.exception(e)
self._view_edit_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
return
self._rescan_config_directory()
@@ -145,7 +152,7 @@ class ViewEditController(QObject):
self._start_search_config_files_async(new_config_directory)
except RuntimeError as e:
logger.exception(e)
self._view_edit_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
def _rescan_config_directory(self) -> None:
configuration: Configuration = self._configuration_manager.configuration
@@ -155,14 +162,14 @@ class ViewEditController(QObject):
configuration.config_directory, constants.RESOURCE_MAPPING_CONFIG_FILE_NAME_SUFFIX)
except FileNotFoundError as e:
logger.exception(e)
self._view_edit_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
return
self._configuration_manager.configuration.config_files = config_files
self._view_edit_page.set_config_files(config_files)
def _reset_page(self) -> None:
self._view_edit_page.hide_notification_frame()
self._view_edit_page.notification_frame.setVisible(False)
self._view_edit_page.set_table_view_page_interactions_enabled(True)
def _save_changes(self) -> None:
@@ -178,14 +185,14 @@ class ViewEditController(QObject):
self._proxy_model.override_all_resources_status(
ResourceMappingAttributesStatus(ResourceMappingAttributesStatus.SUCCESS_STATUS_VALUE,
[ResourceMappingAttributesStatus.SUCCESS_STATUS_VALUE]))
self._view_edit_page.set_notification_frame_text(
self.set_notification_frame_text_sender.emit(
notification_label_text.VIEW_EDIT_PAGE_SAVING_SUCCEED_MESSAGE.format(config_file))
def _search_complete_callback(self) -> None:
self._reset_page()
def _start_search_config_files_async(self, config_directory: str) -> None:
self._view_edit_page.set_notification_page_text(notification_label_text.NOTIFICATION_LOADING_MESSAGE)
self.set_notification_page_frame_text_sender.emit(notification_label_text.NOTIFICATION_LOADING_MESSAGE)
self._view_edit_page.set_current_main_view_index(ViewEditPageConstants.NOTIFICATION_PAGE_INDEX)
self._config_file_json_source.clear()
self._table_view.reset_view()
@@ -202,7 +209,7 @@ class ViewEditController(QObject):
config_directory, constants.RESOURCE_MAPPING_CONFIG_FILE_NAME_SUFFIX)
except FileNotFoundError as e:
logger.exception(e)
self._view_edit_page.set_notification_frame_text(str(e))
self.set_notification_frame_text_sender.emit(str(e))
def _select_config_file(self) -> None:
if self._view_edit_page.config_file_combobox.currentIndex() == -1:
@@ -219,7 +226,7 @@ class ViewEditController(QObject):
self._proxy_model.emit_source_model_layout_changed()
else:
self._view_edit_page.set_table_view_page_interactions_enabled(False)
self._view_edit_page.set_notification_frame_text(
self.set_notification_frame_text_sender.emit(
error_messages.VIEW_EDIT_PAGE_READ_FROM_JSON_FAILED_WITH_UNEXPECTED_FILE_ERROR_MESSAGE.format(config_file_name))
self._view_edit_page.set_current_main_view_index(ViewEditPageConstants.TABLE_VIEW_PAGE_INDEX)
@@ -262,13 +269,13 @@ class ViewEditController(QObject):
invalid_details))
invalid_proxy_rows: List[int] = self._proxy_model.map_from_source_rows(list(invalid_sources.keys()))
self._view_edit_page.set_notification_frame_text(
self.set_notification_frame_text_sender.emit(
error_messages.VIEW_EDIT_PAGE_SAVING_FAILED_WITH_INVALID_ROW_ERROR_MESSAGE.format(invalid_proxy_rows))
return False
return True
@Slot(list)
def add_import_resources(self, resources: List[BasicResourceAttributes]) -> None:
def add_import_resources_receiver(self, resources: List[BasicResourceAttributes]) -> None:
resource: BasicResourceAttributes
for resource in resources:
resource_builder: ResourceMappingAttributesBuilder = ResourceMappingAttributesBuilder() \
@@ -51,4 +51,5 @@ class ControllerManager(object):
logger.info("Setting up ViewEdit and ImportResource controllers ...")
self._view_edit_controller.setup()
self._import_resources_controller.setup()
self._import_resources_controller.add_import_resources.connect(self._view_edit_controller.add_import_resources)
self._import_resources_controller.add_import_resources_sender.connect(
self._view_edit_controller.add_import_resources_receiver)
@@ -32,7 +32,7 @@ VIEW_EDIT_PAGE_SAVING_SUCCEED_MESSAGE: str = "Config file {} is saved successful
IMPORT_RESOURCES_PAGE_BACK_TEXT: str = "Back"
IMPORT_RESOURCES_PAGE_AWS_SEARCH_TYPE_TEXT: str = "AWS Resource Type"
IMPORT_RESOURCES_PAGE_SEARCH_TEXT: str = " Search"
IMPORT_RESOURCES_PAGE_SEARCH_TEXT: str = "Search"
IMPORT_RESOURCES_PAGE_IMPORT_TEXT: str = "Import"
IMPORT_RESOURCES_PAGE_SEARCH_PLACEHOLDER_TEXT: str = "Search for resources by Type or Name/ID"
@@ -25,10 +25,10 @@ VIEW_EDIT_PAGE_FOOTER_AREA_HEIGHT: int = 50
VIEW_EDIT_PAGE_MARGIN_TOPBOTTOM: int = 10
# header area
CONFIG_FILE_LABEL_WIDTH: int = 70
CONFIG_FILE_LABEL_WIDTH: int = 65
CONFIG_FILE_COMBOBOX_WIDTH: int = 250
CONFIG_LOCATION_LABEL_WIDTH: int = 110
CONFIG_LOCATION_TEXT_WIDTH: int = 190
CONFIG_LOCATION_LABEL_WIDTH: int = 100
CONFIG_LOCATION_TEXT_WIDTH: int = 180
HEADER_AREA_SEPARATOR_WIDTH: int = 5
# center area
@@ -9,32 +9,56 @@ remove or modify any license notices. This file is distributed on an "AS IS" BAS
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
"""
from argparse import (ArgumentParser, Namespace)
import logging
import sys
from PySide2.QtCore import Qt
from PySide2.QtWidgets import QApplication
from manager.configuration_manager import ConfigurationManager
from manager.controller_manager import ControllerManager
from manager.thread_manager import ThreadManager
from manager.view_manager import ViewManager
from style import azqtcomponents_resources
from utils import environment_utils
from utils import file_utils
# arguments setup
argument_parser: ArgumentParser = ArgumentParser()
argument_parser.add_argument('--binaries_path', help='Path to QT Binaries necessary for PySide.')
argument_parser.add_argument('--debug', action='store_true', help='Execute on debug mode.')
arguments: Namespace = argument_parser.parse_args()
# logging setup
logging.basicConfig(filename="resource_mapping_tool.log", filemode='w', level=logging.INFO,
logging_level: int = logging.INFO
if arguments.debug:
logging_level = logging.DEBUG
logging_path: str = file_utils.join_path(file_utils.get_parent_directory_path(__file__),
'resource_mapping_tool.log')
logging.basicConfig(filename=logging_path, filemode='w', level=logging_level,
format='%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)
logging.getLogger('s3transfer').setLevel(logging.CRITICAL)
logging.getLogger('urllib3').setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)
if __name__ == "__main__":
if arguments.binaries_path and not environment_utils.is_qt_linked():
logger.info("Setting up Qt environment ...")
environment_utils.setup_qt_environment(arguments.binaries_path)
try:
logger.info("Importing tool required modules ...")
from PySide2.QtCore import Qt
from PySide2.QtWidgets import QApplication
from manager.configuration_manager import ConfigurationManager
from manager.controller_manager import ControllerManager
from manager.thread_manager import ThreadManager
from manager.view_manager import ViewManager
from style import azqtcomponents_resources
except ImportError as e:
logger.error(f"Failed to import module [{e.name}] {e}")
environment_utils.cleanup_qt_environment()
exit(-1)
QApplication.setAttribute(Qt.AA_EnableHighDpiScaling)
QApplication.setAttribute(Qt.AA_UseHighDpiPixmaps)
app: QApplication = QApplication(sys.argv)
app.aboutToQuit.connect(environment_utils.cleanup_qt_environment)
try:
style_sheet_path: str = file_utils.join_path(file_utils.get_parent_directory_path(__file__),
@@ -62,5 +86,5 @@ if __name__ == "__main__":
controller_manager.setup()
view_manager.show()
sys.exit(app.exec_())
@@ -62,7 +62,8 @@ class TestImportResourcesController(TestCase):
self._mocked_proxy_model: MagicMock = self._mocked_tree_view.resource_proxy_model
self._test_import_resources_controller: ImportResourcesController = ImportResourcesController()
self._test_import_resources_controller.add_import_resources = MagicMock()
self._test_import_resources_controller.add_import_resources_sender = MagicMock()
self._test_import_resources_controller.set_notification_frame_text_sender = MagicMock()
self._test_import_resources_controller.setup()
def test_reset_page_resetting_page_with_expected_state(self) -> None:
@@ -116,7 +117,7 @@ class TestImportResourcesController(TestCase):
self._mocked_import_resources_page.typed_resources_search_button.clicked.connect.call_args[0]
mocked_call_args[0]() # triggering search_button connected function
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_tree_view.reset_view.assert_not_called()
self._mocked_import_resources_page.set_current_main_view_index.assert_not_called()
@@ -170,7 +171,7 @@ class TestImportResourcesController(TestCase):
mock_aws_utils.list_cloudformation_stacks.assert_called_once_with(
TestImportResourcesController._expected_region)
mock_aws_utils.list_cloudformation_stack_resources.assert_not_called()
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.load_resource.assert_not_called()
self._mocked_proxy_model.emit_source_model_layout_changed.assert_called_once()
self._mocked_import_resources_page.set_current_main_view_index.assert_called_with(
@@ -195,7 +196,7 @@ class TestImportResourcesController(TestCase):
TestImportResourcesController._expected_region)
mock_aws_utils.list_cloudformation_stack_resources.assert_called_once_with(
TestImportResourcesController._expected_cfn_stack_name, TestImportResourcesController._expected_region)
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.load_resource.assert_not_called()
self._mocked_proxy_model.emit_source_model_layout_changed.assert_called_once()
self._mocked_import_resources_page.set_current_main_view_index.assert_called_with(
@@ -258,8 +259,8 @@ class TestImportResourcesController(TestCase):
self._mocked_import_resources_page.cfn_stacks_import_button.clicked.connect.call_args[0]
mocked_call_args[0]() # triggering cfn_stacks_import_button connected function
self._test_import_resources_controller.add_import_resources.emit.assert_not_called()
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.add_import_resources_sender.emit.assert_not_called()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
def test_page_cfn_stacks_import_button_emit_signal_with_expected_resources_and_switch_to_expected_page(self) -> None:
self._mocked_proxy_model.deduplicate_selected_import_resources.return_value = \
@@ -268,7 +269,7 @@ class TestImportResourcesController(TestCase):
self._mocked_import_resources_page.cfn_stacks_import_button.clicked.connect.call_args[0]
mocked_call_args[0]() # triggering cfn_stacks_import_button connected function
self._test_import_resources_controller.add_import_resources.emit.assert_called_once_with(
self._test_import_resources_controller.add_import_resources_sender.emit.assert_called_once_with(
[TestImportResourcesController._expected_lambda_resource])
self._mocked_view_manager.switch_to_view_edit_page.assert_called_once()
self._mocked_tree_view.reset_view.assert_called_once()
@@ -329,7 +330,7 @@ class TestImportResourcesController(TestCase):
mock_aws_utils.list_lambda_functions.assert_called_once_with(
TestImportResourcesController._expected_region)
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.load_resource.assert_not_called()
self._mocked_proxy_model.emit_source_model_layout_changed.assert_called_once()
self._mocked_import_resources_page.set_current_main_view_index.assert_called_with(
@@ -348,7 +349,7 @@ class TestImportResourcesController(TestCase):
mocked_async_call_args: call = mock_thread_manager.get_instance.return_value.start.call_args[0]
mocked_async_call_args[0].run() # triggering async function
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.load_resource.assert_not_called()
self._mocked_proxy_model.emit_source_model_layout_changed.assert_called_once()
self._mocked_import_resources_page.set_current_main_view_index.assert_called_with(
@@ -383,8 +384,8 @@ class TestImportResourcesController(TestCase):
self._mocked_import_resources_page.typed_resources_import_button.clicked.connect.call_args[0]
mocked_call_args[0]() # triggering typed_resources_import_button connected function
self._test_import_resources_controller.add_import_resources.emit.assert_not_called()
self._mocked_import_resources_page.set_notification_frame_text.assert_called_once()
self._test_import_resources_controller.add_import_resources_sender.emit.assert_not_called()
self._test_import_resources_controller.set_notification_frame_text_sender.emit.assert_called_once()
def test_page_typed_resources_import_button_emit_signal_with_expected_resources_and_switch_to_expected_page(self) -> None:
self._mocked_proxy_model.deduplicate_selected_import_resources.return_value = \
@@ -393,7 +394,7 @@ class TestImportResourcesController(TestCase):
self._mocked_import_resources_page.typed_resources_import_button.clicked.connect.call_args[0]
mocked_call_args[0]() # triggering typed_resources_import_button connected function
self._test_import_resources_controller.add_import_resources.emit.assert_called_once_with(
self._test_import_resources_controller.add_import_resources_sender.emit.assert_called_once_with(
[TestImportResourcesController._expected_lambda_resource])
self._mocked_view_manager.switch_to_view_edit_page.assert_called_once()
self._mocked_tree_view.reset_view.assert_called_once()
@@ -62,10 +62,12 @@ class TestViewEditController(TestCase):
self._mocked_proxy_model: MagicMock = self._mocked_table_view.resource_proxy_model
self._test_view_edit_controller: ViewEditController = ViewEditController()
self._test_view_edit_controller.set_notification_frame_text_sender = MagicMock()
self._test_view_edit_controller.set_notification_page_frame_text_sender = MagicMock()
self._test_view_edit_controller.setup()
def test_add_import_resources_expected_resource_gets_loaded_into_model(self) -> None:
self._test_view_edit_controller.add_import_resources([TestViewEditController._expected_resource])
self._test_view_edit_controller.add_import_resources_receiver([TestViewEditController._expected_resource])
self._mocked_proxy_model.add_resource.assert_called_once()
mocked_call_args: call = self._mocked_proxy_model.add_resource.call_args[0] # mock call args index is 0
@@ -128,7 +130,7 @@ class TestViewEditController(TestCase):
self._mocked_proxy_model.emit_source_model_layout_changed.assert_not_called()
self._mocked_proxy_model.load_resource.assert_not_called()
self._mocked_view_edit_page.set_table_view_page_interactions_enabled.assert_called_with(False)
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_view_edit_page.set_current_main_view_index.assert_called_with(
ViewEditPageConstants.TABLE_VIEW_PAGE_INDEX)
@@ -189,7 +191,7 @@ class TestViewEditController(TestCase):
mock_json_utils.validate_json_dict_according_to_json_schema.assert_called_once_with({})
mock_json_utils.convert_json_dict_to_resources.assert_not_called()
self._mocked_proxy_model.load_resource.assert_not_called()
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_view_edit_page.set_table_view_page_interactions_enabled.assert_called_with(False)
@patch("controller.view_edit_controller.file_utils")
@@ -239,7 +241,7 @@ class TestViewEditController(TestCase):
self._mocked_view_edit_page.config_file_combobox.currentText.assert_called_once()
self._mocked_table_view.reset_view.assert_called_once()
self._mocked_proxy_model.override_resource_status.assert_called_once()
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.emit_source_model_layout_changed.assert_has_calls([call(), call()])
self._mocked_view_edit_page.set_current_main_view_index.assert_called_with(
ViewEditPageConstants.TABLE_VIEW_PAGE_INDEX)
@@ -275,7 +277,7 @@ class TestViewEditController(TestCase):
mocked_call_args[0]() # triggering config_location_button connected function
mock_file_dialog.getExistingDirectory.assert_called_once()
self._mocked_view_edit_page.set_notification_page_text.assert_called_with(
self._test_view_edit_controller.set_notification_page_frame_text_sender.emit.assert_called_with(
notification_label_text.NOTIFICATION_LOADING_MESSAGE)
self._mocked_view_edit_page.set_current_main_view_index.assert_called_with(
ViewEditPageConstants.NOTIFICATION_PAGE_INDEX)
@@ -303,7 +305,7 @@ class TestViewEditController(TestCase):
expected_new_config_directory, constants.RESOURCE_MAPPING_CONFIG_FILE_NAME_SUFFIX)
assert self._mocked_configuration_manager.configuration.config_files == []
self._mocked_view_edit_page.set_config_files.assert_called_with([])
self._mocked_view_edit_page.set_notification_page_text.assert_called_once_with(
self._test_view_edit_controller.set_notification_page_frame_text_sender.emit.assert_called_once_with(
notification_label_text.NOTIFICATION_LOADING_MESSAGE)
@patch("controller.view_edit_controller.ThreadManager")
@@ -325,7 +327,7 @@ class TestViewEditController(TestCase):
expected_new_config_directory, constants.RESOURCE_MAPPING_CONFIG_FILE_NAME_SUFFIX)
assert self._mocked_configuration_manager.configuration.config_files == []
self._mocked_view_edit_page.set_config_files.assert_called_with([])
self._mocked_view_edit_page.set_notification_page_text.assert_called_once_with(
self._test_view_edit_controller.set_notification_page_frame_text_sender.emit.assert_called_once_with(
notification_label_text.NOTIFICATION_LOADING_MESSAGE)
@patch("controller.view_edit_controller.ThreadManager")
@@ -348,7 +350,7 @@ class TestViewEditController(TestCase):
expected_new_config_directory, constants.RESOURCE_MAPPING_CONFIG_FILE_NAME_SUFFIX)
assert self._mocked_configuration_manager.configuration.config_files == expected_new_config_files
self._mocked_view_edit_page.set_config_files.assert_called_with(expected_new_config_files)
self._mocked_view_edit_page.set_notification_page_text.assert_called_once_with(
self._test_view_edit_controller.set_notification_page_frame_text_sender.emit.assert_called_once_with(
notification_label_text.NOTIFICATION_LOADING_MESSAGE)
def test_page_add_row_button_expected_resource_gets_loaded_into_model(self) -> None:
@@ -395,7 +397,7 @@ class TestViewEditController(TestCase):
mocked_call_args[0]() # triggering save_changes_button connected function
self._mocked_proxy_model.override_resource_status.assert_called_once()
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.override_all_resources_status.assert_not_called()
@patch("controller.view_edit_controller.json_utils")
@@ -451,7 +453,7 @@ class TestViewEditController(TestCase):
mock_json_utils.convert_resources_to_json_dict.assert_called_once()
mock_json_utils.write_into_json_file.assert_called_once_with(
TestViewEditController._expected_config_file_full_path, expected_json_dict)
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
self._mocked_proxy_model.override_all_resources_status.assert_not_called()
def test_page_search_filter_input_invoke_proxy_model_with_expected_filter_text(self) -> None:
@@ -498,7 +500,7 @@ class TestViewEditController(TestCase):
mock_file_utils.join_path.assert_called_once()
mock_json_utils.create_empty_resource_mapping_file.assert_called_once()
mock_file_utils.find_files_with_suffix_under_directory.assert_not_called()
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
@patch("controller.view_edit_controller.file_utils")
def test_page_rescan_button_post_notification_when_find_files_throw_exception(
@@ -509,4 +511,4 @@ class TestViewEditController(TestCase):
mocked_call_args[0]() # triggering rescan_button connected function
mock_file_utils.find_files_with_suffix_under_directory.assert_called_once()
self._mocked_view_edit_page.set_notification_frame_text.assert_called_once()
self._test_view_edit_controller.set_notification_frame_text_sender.emit.assert_called_once()
@@ -55,5 +55,5 @@ class TestControllerManager(TestCase):
TestControllerManager._expected_controller_manager.setup()
mocked_view_edit_controller.setup.assert_called_once()
mocked_import_resources_controller.setup.assert_called_once()
mocked_import_resources_controller.add_import_resources.connect.assert_called_once_with(
mocked_view_edit_controller.add_import_resources)
mocked_import_resources_controller.add_import_resources_sender.connect.assert_called_once_with(
mocked_view_edit_controller.add_import_resources_receiver)
@@ -18,7 +18,7 @@ from manager.view_manager import (ViewManager, ViewManagerConstants)
class TestViewManager(TestCase):
"""
ThreadManager unit test cases
ViewManager unit test cases
"""
_mock_import_resources_page: MagicMock
_mock_view_edit_page: MagicMock
@@ -36,6 +36,8 @@ class TestViewManager(TestCase):
main_window_patcher: patch = patch("manager.view_manager.QMainWindow")
cls._mock_main_window = main_window_patcher.start()
window_icon_patcher: patch = patch("manager.view_manager.QPixmap")
window_icon_patcher.start()
stacked_pages_patcher: patch = patch("manager.view_manager.QStackedWidget")
cls._mock_stacked_pages = stacked_pages_patcher.start()
@@ -0,0 +1,44 @@
"""
All or portions of this file Copyright (c) Amazon.com, Inc. or its affiliates or
its licensors.
For complete copyright and license terms please see the LICENSE at the root of this
distribution (the "License"). All use of this software is governed by the License,
or, if provided, by the license below or the license accompanying this file. Do not
remove or modify any license notices. This file is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
"""
from typing import List
from unittest import TestCase
from unittest.mock import (ANY, call, MagicMock, patch)
from model import constants
from model.basic_resource_attributes import (BasicResourceAttributes, BasicResourceAttributesBuilder)
from utils import environment_utils
class TestEnvironmentUtils(TestCase):
"""
environment utils unit test cases
"""
def setUp(self) -> None:
os_environ_patcher: patch = patch("os.environ")
self.addCleanup(os_environ_patcher.stop)
self._mock_os_environ: MagicMock = os_environ_patcher.start()
os_pathsep_patcher: patch = patch("os.pathsep")
self.addCleanup(os_pathsep_patcher.stop)
self._mock_os_pathsep: MagicMock = os_pathsep_patcher.start()
def test_setup_qt_environment_global_flag_is_set(self) -> None:
environment_utils.setup_qt_environment("dummy")
self._mock_os_environ.copy.assert_called_once()
self._mock_os_pathsep.join.assert_called_once()
assert environment_utils.is_qt_linked() is True
def test_cleanup_qt_environment_global_flag_is_set(self) -> None:
environment_utils.setup_qt_environment("dummy")
assert environment_utils.is_qt_linked() is True
environment_utils.cleanup_qt_environment()
assert environment_utils.is_qt_linked() is False
@@ -30,10 +30,6 @@ class TestFileUtils(TestCase):
self.addCleanup(path_patcher.stop)
self._mock_path: MagicMock = path_patcher.start()
windows_path_patcher: patch = patch("pathlib.WindowsPath")
self.addCleanup(windows_path_patcher.stop)
self._mock_windows_path: MagicMock = windows_path_patcher.start()
def test_check_path_exists_returns_true(self) -> None:
mocked_path: MagicMock = self._mock_path.return_value
mocked_path.exists.return_value = True
@@ -105,12 +101,35 @@ class TestFileUtils(TestCase):
assert not actual_files
def test_join_path_return_expected_result(self) -> None:
mocked_windows_path: MagicMock = self._mock_windows_path.return_value
mocked_path: MagicMock = self._mock_path.return_value
expected_join_path_name: str = f"{TestFileUtils._expected_path_name}{TestFileUtils._expected_file_name}"
mocked_windows_path.joinpath.return_value = expected_join_path_name
mocked_path.joinpath.return_value = expected_join_path_name
actual_join_path_name: str = file_utils.join_path(TestFileUtils._expected_path_name,
TestFileUtils._expected_file_name)
self._mock_windows_path.assert_called_once_with(TestFileUtils._expected_path_name)
mocked_windows_path.joinpath.assert_called_once_with(TestFileUtils._expected_file_name)
self._mock_path.assert_called_once_with(TestFileUtils._expected_path_name)
mocked_path.joinpath.assert_called_once_with(TestFileUtils._expected_file_name)
assert actual_join_path_name == expected_join_path_name
def test_normalize_file_path_return_empty_when_input_is_empty(self) -> None:
actual_normalized_path: str = file_utils.normalize_file_path("")
assert actual_normalized_path == ""
def test_normalize_file_path_return_expected_result(self) -> None:
mocked_path: MagicMock = self._mock_path.return_value
expected_resolve_path: str = TestFileUtils._expected_path_name
mocked_path.resolve.return_value = expected_resolve_path
actual_resolve_path: str = file_utils.normalize_file_path("dummy")
self._mock_path.assert_called_once()
mocked_path.resolve.assert_called_once()
assert actual_resolve_path == expected_resolve_path
def test_normalize_file_path_return_empty_when_exception_raised(self) -> None:
mocked_path: MagicMock = self._mock_path.return_value
mocked_path.resolve.side_effect = RuntimeError()
actual_resolve_path: str = file_utils.normalize_file_path("dummy")
self._mock_path.assert_called_once()
mocked_path.resolve.assert_called_once()
assert actual_resolve_path == ""
@@ -44,15 +44,26 @@ class AWSConstants(object):
S3_SERVICE_NAME: str = "s3"
def _close_client_connection(client: BaseClient) -> None:
session: boto3.session.Session = client._endpoint.http_session
managers: List[object] = [session._manager, *session._proxy_managers.values()]
for manager in managers:
manager.clear()
def _initialize_boto3_aws_client(service: str, region: str = "") -> BaseClient:
if region:
return boto3.client(service, region_name=region)
boto3_client: BaseClient = boto3.client(service, region_name=region)
else:
return boto3.client(service)
boto3_client: BaseClient = boto3.client(service)
boto3_client.meta.events.register(
f"after-call.{service}.*", lambda **kwargs: _close_client_connection(boto3_client)
)
return boto3_client
def get_default_account_id() -> str:
sts_client: BaseClient = boto3.client(AWSConstants.STS_SERVICE_NAME)
sts_client: BaseClient = _initialize_boto3_aws_client(AWSConstants.STS_SERVICE_NAME)
try:
return sts_client.get_caller_identity()["Account"]
except ClientError as error:
@@ -65,7 +76,7 @@ def get_default_region() -> str:
if region:
return region
sts_client: BaseClient = boto3.client(AWSConstants.STS_SERVICE_NAME)
sts_client: BaseClient = _initialize_boto3_aws_client(AWSConstants.STS_SERVICE_NAME)
region = sts_client.meta.region_name
if region:
return region
@@ -0,0 +1,70 @@
"""
All or portions of this file Copyright (c) Amazon.com, Inc. or its affiliates or
its licensors.
For complete copyright and license terms please see the LICENSE at the root of this
distribution (the "License"). All use of this software is governed by the License,
or, if provided, by the license below or the license accompanying this file. Do not
remove or modify any license notices. This file is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
"""
import logging
import os
from typing import Dict
from utils import file_utils
"""
Environment Utils provide functions to setup python environment libs for resource mapping tool
"""
logger = logging.getLogger(__name__)
qt_binaries_linked: bool = False
old_os_env: Dict[str, str] = os.environ.copy()
def setup_qt_environment(bin_path: str) -> None:
"""
Setup Qt binaries for o3de python runtime environment
:param bin_path: The path of Qt binaries
"""
if is_qt_linked():
logger.info("Qt binaries have already been linked, skip Qt setup")
return
global old_os_env
old_os_env = os.environ.copy()
binaries_path: str = file_utils.normalize_file_path(bin_path)
os.environ["QT_PLUGIN_PATH"] = binaries_path
path = os.environ['PATH']
new_path = os.pathsep.join([binaries_path, path])
os.environ['PATH'] = new_path
global qt_binaries_linked
qt_binaries_linked = True
def is_qt_linked() -> bool:
"""
Check whether Qt binaries have been linked in o3de python runtime environment
:return: True if Qt binaries have been linked; False if not
"""
return qt_binaries_linked
def cleanup_qt_environment() -> None:
"""
Clean up the linked Qt binaries in o3de python runtime environment
"""
if not is_qt_linked():
logger.info("Qt binaries have not been linked, skip Qt uninstall")
return
global old_os_env
if old_os_env.get("QT_PLUGIN_PATH"):
old_os_env.pop("QT_PLUGIN_PATH")
os.environ = old_os_env
global qt_binaries_linked
qt_binaries_linked = False
@@ -20,8 +20,8 @@ path, check file existence, etc
logger = logging.getLogger(__name__)
def check_path_exists(full_path: str) -> bool:
return pathlib.Path(full_path).exists()
def check_path_exists(file_path: str) -> bool:
return pathlib.Path(file_path).exists()
def get_current_directory_path() -> str:
@@ -42,8 +42,16 @@ def find_files_with_suffix_under_directory(dir_path: str, suffix: str) -> List[s
if matched_path.is_file():
results.append(str(matched_path.name))
return results
def join_path(dir_path: str, file_name: str) -> str:
# TODO: expand usage to support Mac and Linux
return str(pathlib.WindowsPath(dir_path).joinpath(file_name))
def normalize_file_path(file_path: str) -> str:
if file_path:
try:
return str(pathlib.Path(file_path).resolve(True))
except (FileNotFoundError, RuntimeError):
logger.warning(f"Failed to normalize file path {file_path}, return empty string instead")
return ""
def join_path(this_path: str, other_path: str) -> str:
return str(pathlib.Path(this_path).joinpath(other_path))
@@ -9,6 +9,7 @@ remove or modify any license notices. This file is distributed on an "AS IS" BAS
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
"""
from PySide2.QtCore import Slot
from PySide2.QtGui import (QIcon, QPixmap)
from PySide2.QtWidgets import (QFrame, QHBoxLayout, QLabel, QLayout, QLineEdit, QPushButton,
QSizePolicy, QWidget)
@@ -58,5 +59,7 @@ class NotificationFrame(QFrame):
self.setVisible(False)
def set_text(self, text: str) -> None:
@Slot(str)
def set_frame_text_receiver(self, text: str) -> None:
self._title_label.setText(text)
self.setVisible(True)
@@ -136,7 +136,6 @@ class ImportResourcesPage(QWidget):
self._back_button.setObjectName("Secondary")
self._back_button.setText(f" {notification_label_text.IMPORT_RESOURCES_PAGE_BACK_TEXT}")
self._back_button.setIcon(QIcon(":/Breadcrumb/img/UI20/Breadcrumb/arrow_left-default.svg"))
self._back_button.setFlat(True)
self._back_button.setMinimumSize(view_size_constants.BACK_BUTTON_WIDTH,
view_size_constants.INTERACTION_COMPONENT_HEIGHT)
header_area_layout.addWidget(self._back_button)
@@ -325,6 +324,10 @@ class ImportResourcesPage(QWidget):
def search_version(self) -> str:
return self._search_version
@property
def notification_frame(self) -> NotificationFrame:
return self._notification_frame
@search_version.setter
def search_version(self, new_search_version: str) -> None:
self._search_version = new_search_version
@@ -343,10 +346,3 @@ class ImportResourcesPage(QWidget):
def set_current_main_view_index(self, index: int) -> None:
"""Switch main view page based on given index"""
self._stacked_pages.setCurrentIndex(index)
def hide_notification_frame(self) -> None:
self._notification_frame.setVisible(False)
def set_notification_frame_text(self, text: str) -> None:
self._notification_frame.set_text(text)
self._notification_frame.setVisible(True)
@@ -410,6 +410,14 @@ class ViewEditPage(QWidget):
def rescan_button(self) -> QPushButton:
return self._rescan_button
@property
def notification_frame(self) -> NotificationFrame:
return self._notification_frame
@property
def notification_page_frame(self) -> NotificationFrame:
return self._notification_page_frame
def set_current_main_view_index(self, index: int) -> None:
"""Switch main view page based on given index"""
if index == ViewEditPageConstants.NOTIFICATION_PAGE_INDEX:
@@ -436,11 +444,13 @@ class ViewEditPage(QWidget):
self._config_file_combobox.setCurrentIndex(-1)
if config_files:
self._notification_page_frame.set_text(notification_label_text.VIEW_EDIT_PAGE_SELECT_CONFIG_FILE_MESSAGE)
self._notification_page_frame.set_frame_text_receiver(
notification_label_text.VIEW_EDIT_PAGE_SELECT_CONFIG_FILE_MESSAGE)
self._create_new_button.setVisible(False)
self._rescan_button.setVisible(False)
else:
self._notification_page_frame.set_text(notification_label_text.VIEW_EDIT_PAGE_NO_CONFIG_FILE_FOUND_MESSAGE)
self._notification_page_frame.set_frame_text_receiver(
notification_label_text.VIEW_EDIT_PAGE_NO_CONFIG_FILE_FOUND_MESSAGE)
self._create_new_button.setVisible(True)
self._rescan_button.setVisible(True)
@@ -455,16 +465,6 @@ class ViewEditPage(QWidget):
self._config_location_text.setText(elided_text)
self._config_location_text.setToolTip(config_location)
def set_notification_page_text(self, text: str) -> None:
self._notification_page_frame.set_text(text)
def hide_notification_frame(self) -> None:
self._notification_frame.setVisible(False)
def set_notification_frame_text(self, text: str) -> None:
self._notification_frame.set_text(text)
self._notification_frame.setVisible(True)
def set_table_view_page_interactions_enabled(self, enabled: bool) -> None:
self._table_view_page.setEnabled(enabled)
self._save_changes_button.setEnabled(enabled)