This removes mostly benign (but noisy) messages about "Failed to find builder dependency".
Changed AssetUtils::GetPossibleDepenencyPaths to return all possible source paths, rather than stopping when one is found. This function is now used to report a list of all possible source dependencies, so that CreateJobs will get called by the AP whenever a file shows up at one of those locations. If a file was missing before and then appears, this will cause the builders to wake up and add the appropriate job dependencies on the new files. In ShaderVariantAssetBuilder and MaterialBuilder, we now use GetPossibleDepenencyPaths to report source dependencies rather than job dependencies. We only report a job dependency when the actual source file has been identified. This should all now be consistent with the intended design of the AP's dependency systems (the prior approach was a hack based on misunderstanding of what source dependencies are).
SrgLayoutBuilder's change is a bit tricky. The above changes did not fix all of the "Failed to find builder dependency" messages because AzslBuilder sometimes skips particular files in CreateJobs. When this happens, it is invalid to report an AzslBuilder job dependency on that file. So I copied the same conditional code that is used to skip an azsli file in CreateJobs, and used that to skip AddAzslBuilderJobDependency() in SrgLayoutBuilder as well.
With all these changes combined, I do think we've solved the issue where jobs fail to evict outdated jobs, as described in ATOM-15134. However, we are not yet seeing the iteration time improvements we were hoping for. Before these changes I was seeing roughly a 0.5 minute delay for the initial change, and a 2 minute delay for a subsequent change. With these changes it's more like 0.5 mibutes and 1.5 minutes. It appears that the AP scan is being starved by all the AssetBuilder processing going on, and perhaps IO contention. I suspect that this will be greatly improved on the development branch where we no longer have AzslBuilder and SrgLayoutBuilder slowing things down.
ATOM-15136 Builder dependency errors reported in mainline
ATOM-15134 Replace GetPossibleDepenencyPaths Approach with Source Dependencies
* [ATOM-15618] Shader Build Pipeline: Add UnitTest To Validate Shader Compiler
Argument Processing Introduced With The New Supervariant System
- Added new test suite in
Gems/Atom/Asset/Shader/Code/Tests/SupervariantCmdArgumentTests.cpp
- Refactored and improved the previously existing classes:
GlobalBuildOptions, PreprocessorOptions and ShaderCompilerArguments
to work well with the new ShaderSourceData::SupervariantInfo.
- Moved command line argument processing function out of
ShaderCompilerArguments and into its own utility namespace in
Atom/RHI.Edit/Utils.h
Signed-off-by: garrieta <garrieta@amazon.com>
* [ATOM-15276] Shader Build Pipeline: Add Shader Supervariant System.
Added ShaderAssetBuilder2 & ShaderVariantAssetBuilder2.
Added ShaderAsset2, ShaderVariantAsset2.
Eventually they will be the only builders. AzslBuilder &
SrgLayoutBuilder will be removed.
ShaderResourceGroupAsset will be removed. ShaderAssetBuilder &
ShaderVariantAssetBuilder will be replaced.
Signed-off-by: garrieta <garrieta@amazon.com>
* LYN-2537 Moved the Engine and Editor folder to be within the EngineAssets folder
* Fixed Documentation in bootstrap.cfg to correct the path to the user project specific registry file
* Adding a newline to the output of AssetCatalog 'Registering asset..., but type is not set' message
* Updating the AssetProcessorPlatformConfig.setreg Scan Folder to detect
the @ENGINEROOT@/EngineAssets/Engine path for engine runtime assets and
@ENGINEROOT@/EngineAssets/Editor path for engine tool assets
* Updating references to Icons and other assets to account for moving the
Engine and Editor folder under a single EngineAssets folder
* Moving the Engine Settings Registry folder from Engine/Registry -> Registry
* Removed the LY_PROJECT_CMAKE_PATH define as it is not portable to other locations. It is hard coded to the project location that was used for the CMake configuration. Furthermore it paths with backslashes within it are treated as escape characters and not a path separator
* Updated the LyTestTools asset_processor.py script to copy the exclude.filetag from the EngineAssets/Engine directory now
* Fixed Atom Shader Preprocessing when running using an External Project
* Updated the TSGenerateAction.cpp to fix the build error with using a renamed variable
* Updated the Install_Common.cmake ly_setup_others function to install the
EngineAssets directory and the each of the Gem's Assets directory while
maintaining the relative directory structure to the Engine Root
Also updated the install step to install the Registry folder at the
engine root
* Fixed the copying of the Registry folder to be in the install root, instead of under a second 'Registry' folder
* Moving the AssetProcessorPlatformConfig.setreg file over to the Registry folder
* Updated the LyTestTools and C++ code to point that the new location of
the AssetProcessorPlatformConfig.setreg file inside of the Registry
folder
* Renamed Test AssetProcessor*Config.ini files to have the .setreg extension
* Converted the AssetProcessor test setreg files from ini format to json
format using the SerializeContextTools convert-ini command
* Updated the AssetProcessor CMakeLists.txt to copy over the test setreg files to the build folder
* Updated the assetprocessor test file list to point at the renamed AsssetProcessor*Config setreg filenames
* Removed the Output Prefix code from the AssetProcessor. The complexity that it brought to the AP code is not needed, as users can replicate the behavior by just moving there assets underneath a another folder, underneath the scan folder
* Adding back support to read the AssetProcessorPlatformConfig.setreg file from the asset root. This is only needed for C++ UnitTests as they run in an environment where the accessing the Engine Settings Registry is not available
* Updating the Install_common.cmake logic to copy any "Assets" folder to
the install layout.
The Script has also been updated to copy over the "Assets" folder in the
Engine Root to the install layout instead of an "EngineAssets" folder
* Updating References to EngineAssets source asset folder in code to be the Assets source folder
* Moved the Engine Source Asset folder of 'EngineAssets' to a new folder name of 'Assets'. This is inline with the naming scheme we use for Gem asset folders
* Adding the EngineFinder.cmake to the AutomatedTesting project to allow it to work in a project centric manner
* Updating the LyTestTools copy_assets_to_project function to be able to copy assets with folders to the temporary project root
Fixed an issue in LyTestTools where the temporary log directory could have shutil.rmtree being called twice on it leading to an exception which fails an automated test
Updated the asset_procesor_gui_tests_2 AddScanFolder test to not use the
output prefix, but instead place the source asset root into a
subdirectory
* Correct the AssetProcessorPlatformConfig Scan Folders for the EngineAssets directory to point at the Assets directory
* Updated the asset procesor batch dependency test scan folder to point at the 'Assets' folder instead of 'EngineAssets'
ATOM-14040 Add Support for Cavity Maps
Replace the "Ambient Occlusion" property groups with "Occlusion" property groups that contain settings for "Diffuse AO" and "Specular Cavity" maps.
Pre-multiplied diffuseAmbientOcclusion into the m_albedo.rgb gbuffer, making room for specularOcclusion to be stored in m_albedo.a
Added new render pipeline connections to connect the albedo gbuffer to the fullscreen reflection passes.
Also fixed ATOM-15297 "StandardPBR_ForwardPass.shadervariantlist Job Fails With Obscure Error"
ShaderVariantAssetBuilder::ProcessJob was not handling the should-exit-early flag, described at ValidateShaderVariantListLocation(). It was assuming that this flag would be set only during error conditions, which is not the case. I just had to rearrange the logic to handle deferred errors and should-exit-early separately.
Testing:
The problematic job now finishes successfully. I also edited the shadervariantlist file to give it a bad path to the shader and confirmed that we still get the "Error during CreateJobs" message.
Steve Pham
Chris Santora
Terry Michaels
galibzon
moudgils
lumberyard-employee-dm
Chris Burel
Chris Santora
garrieta
hershey5045
alexpete