(Continuation) Implemented automation paralellization & standarization (#1718)

Engine improvements/fixes

Fixed behavior that made the editor automated test to be sometimes stuck if lost the focus is lost.
Added support for specifying multiple tests to in batch to the editor, this is achieved by passing --runpythontest with the tests separated by ';'
Added new cmdline argument --project-user-path for overriding the user path. This allows to have multiple editors running writing logs and crash logs in different locations.
Moved responsability of exiting after a test finishes/passes out of ExecuteByFilenameAsTest, callers will use the bool return to know if the test passed.
Editor test batch and parallelization implementation:

Now the external python portion of the editor tests will be specified via test specs which will generate the test. Requiring no code. This is almost a data-driven approach.
Tests can be specified as single tests, parallel, batchable or batchable+parallel
Command line arguments for pytest to override the maximum number of editors, disable parallelization or batching.
Automated tests for testing this new editor testing utility

Signed-off-by: Garcia Ruiz <aljanru@amazon.co.uk>

Co-authored-by: Garcia Ruiz <aljanru@amazon.co.uk>

Tools/LyTestTools/ly_test_tools/_internal/pytest_plugin/editor_test.py

@@ -0,0 +1,43 @@
+"""
+Copyright (c) Contributors to the Open 3D Engine Project.
+For complete copyright and license terms please see the LICENSE at the root of this distribution.
+
+SPDX-License-Identifier: Apache-2.0 OR MIT
+"""
+
+"""
+Utility for specifying an Editor test, supports seamless parallelization and/or batching of tests.
+"""
+
+import pytest
+import inspect
+
+__test__ = False
+
+def pytest_addoption(parser):
+    parser.addoption("--no-editor-batch", action="store_true", help="Don't batch multiple tests in single editor")
+    parser.addoption("--no-editor-parallel", action="store_true", help="Don't run multiple editors in parallel")
+    parser.addoption("--parallel-editors", type=int, action="store", help="Override the number editors to run at the same time")
+
+# Create a custom custom item collection if the class defines pytest_custom_makeitem function
+# This is used for automtically generating test functions with a custom collector
+def pytest_pycollect_makeitem(collector, name, obj):
+    if inspect.isclass(obj):
+        for base in obj.__bases__:
+            if hasattr(base, "pytest_custom_makeitem"):
+                return base.pytest_custom_makeitem(collector, name, obj)
+
+# Add custom modification of items.
+# This is used for adding the runners into the item list
+@pytest.hookimpl(hookwrapper=True)
+def pytest_collection_modifyitems(session, items, config):
+    all_classes = set()
+    for item in items:
+        all_classes.add(item.instance.__class__)
+
+    yield
+
+    for cls in all_classes:
+        if hasattr(cls, "pytest_custom_modify_items"):
+            cls.pytest_custom_modify_items(session, items, config)
+                

Tools/LyTestTools/ly_test_tools/_internal/pytest_plugin/test_tools_fixtures.py

@@ -46,7 +46,6 @@ def pytest_addoption(parser):
                      help="An existing CMake binary output directory which contains the lumberyard executables,"
                           "such as: D:/ly/dev/windows_vs2017/bin/profile/")
 
-
 def pytest_configure(config):
     """
     Save custom CLI options during Pytest configuration, so they are later accessible without using fixtures
@@ -57,6 +56,13 @@ def pytest_configure(config):
     ly_test_tools._internal.pytest_plugin.output_path = _get_output_path(config)
 
 
+def pytest_pycollect_makeitem(collector, name, obj):
+    import inspect
+    if inspect.isclass(obj):
+        for base in obj.__bases__:
+            if hasattr(base, "pytest_custom_makeitem"):
+                return base.pytest_custom_makeitem(collector, name, obj)
+
 def _get_build_directory(config):
     """
     Fetch and verify the cmake build directory CLI arg, without creating an error when unset
@@ -359,6 +365,9 @@ def _workspace(request,  # type: _pytest.fixtures.SubRequest
                ):
     """Separate implementation to call directly during unit tests"""
 
+    # Convert build directory to absolute path in case it was provided as relative path
+    build_directory = os.path.abspath(build_directory)
+
     workspace = helpers.create_builtin_workspace(
         build_directory=build_directory,
         project=project,