Fixed an issue where the shader builder would incorrectly succeed when the .azsl file is missing.
Also renamed some variables for more consistency.
Updated all .shader files accordingly.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
* Changed references to "InlineConstant" to "RootConstant".
Updated AZSLC to version 1.7.34 for mac, linux & windows
Signed-off-by: garrieta <garrieta@amazon.com>
* Update the FileIOAlias naming to make the cache, project root and engine
root paths more clear
The alias of `@root@`, `@assets@`, and `@projectplatformcache@` has been
collapsed to `@projectproductassets@`
The alias of `@devroot@` and `@engroot@` has been collapsed to
`@engroot@`
The alias of `@devassets@` and `@projectroot@` has been collapsed to
`@projectroot@`
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated use of devassets and devroot properties in python
Those properties now use projectroot and engroot
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updating the alias @engroot@ alias path comment in each platform specific LocalFileIO_*.cpp file
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed hardcoded size of 9 for the product asset alias.
The ResolvePath function now just appends the @projectproductassets@
alias with the input path
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Remove duplicate @projectproductassets@ check in ProcessFileTreeRequest
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix for typos in Hydra python test
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated LocalFileIO::Copy call on Windows to use the Unicode aware CopyFileW API
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AWSMetreicsGemAllocatorFixture to properly suppress asset
cache write errors for Test file creation.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed unneeded call to set the @projectproductasstes@ alias at the bottom of the AssetSeedManagerTest SetUp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added a deprecated alias map to the FileIO System
When a deprecated alias is accessed, the FileIO System logs an AZ_Error and indicates the alias that should be used
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated python test scripts to use the projectroot binding
Retrieving the AutomatedTesting project path based on "<devroot>/AutomatedTesting" has been removed.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated references to devroot and devgame within the codebase
The GetAbsoluteDevGameFolderPath functions has been replaced with direct call to AZ::Utils::GetProjectPath
The GetAbsoluteDevRootFolderPath functions has been replaced with direct calls to AZ::Utils::GetEnginePath
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated <engroot>/AutomatedTesting references to projectroot
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Replaced references that assumes the project path is <engroot>/AutomatedTesting with <projectroot> in the AutomatedTesting python test
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Correct casing in emfxworkspace file
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed newly added AppendPathParts function
Removed the Path constructors which accepts a PathIterable instance
The PathIterable isn't safe to return to a user of the Path class as it might be referencing temporary paths supplied via PathView arguments
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed unused parameter warning
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Undid change to the LexicallyProximate function to set the path result to the base path.
It needs to return the *this path if the pathResult is empty
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Moved the LocalFileIO ConvertToAbsolutePath implementations to AZ::Utils
Fixed the ConvertToAbsolutePath implementation for Unix platforms to use a buffer that is size PATH_MAX(4096 on all our supported Unix platforms).
Because the buffer before was AZ::IO::MaxPathLength which as a size of 1024, this was resulting in the Unix `realpath` function smashing the stack when it converts a path to an absolute path that is longer than 1024 characters
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the EditorCore.Tests to attach the AZ Environment to the EditorCore shared library that is statically loaded on launch.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed for DeprecatedAlaisesKeyVisitor Visit function causing the non string_view overloads being hidden causing a hidden base class function warning
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Changed the AWSMetricsGemMock to use a temporary for writing test assets
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the LocalFileIO::ResolvePath function to use HasRootPath to determine if a path starts with a root path such as "D:", "D:/" or "/"
IsAbsolute was not the corect check as the path of "D:" is a relative
path.
To be absolute according to the Windows the path must have a root
directory. i.e "D:/" or "D:\"
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed absolute path comment from LocalFile_UnixLike.cpp and LocalFile_Android.cpp FindFiles implementations
Updated the ConvertToAlias to supply an AZ::IO::FixedMaxPath
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Replaced usage of the @projectproductassets@ alias with @engroot@ when referring to the LmbrCentral source folder in the CopyDependencyBuilderTest and the SeedBuilderTests
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the ScriptCanvas Upgrade Tool to output backed up files to the
Project User Path instead of the engine root
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed whitespacing issues in Application.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Remove unnecessary creation of a FixedMaxPath in the UpgradeTool.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Modified testSeedFolder variable in the SeedBuilderTests to use the
@engroot@ alias instead of @projectproductassets@/.. alias when
referring to the LmbrCentral Gem source path
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated references to the Project Asset Cache in the PythonTests.
Those tests no longer use the logic `azlmbr.paths.projectroot / "Cache" / "pc"` to retrieve a path to the cache root but instead the `azlmbr.paths.projectproductassets` constant
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed the FileIO Deprecated Alias test on Windows
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removing @projectsourceassets@ alias, as it is only used once.
Updated the PhysX EditorSystemComponent.cpp to query the ProjectPath
form the SettingsRegistry.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Replaced @projectproductassets@ alias with @products@
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Rollback changes to the PhysX EditorSystemComponent.cpp
The changes to use the ProjectPath from the SettingsRegistry has been implemented in PR #4497
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* adding Windows/release to PR-validation builds
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* changing trace back to expand to nothing for release
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* typo
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* more fixes
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* fixing some more unused variable cases
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* renaming file in ScriptCanvas that causes a msbuild warning
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* reverting a previous change
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* Add missing 'server' platform identifier for the ShaderBuilder
* Use the current host platform as the fallback platform identifier (and not 'pc')
Signed-off-by: Steve Pham <spham@amazon.com>
* [ATOM-5441] Shader Builders May Fail When Multiple
New Files Are Added
ShaderAssetBuilder::CreateJobs now recursively parses *.azsl files
looking for #include lines and builds the list of source dependencies
using a depth-first algorithm. It was using MCPP before but not anymore
(during CreateJobs).
The new algorithm may over prescribe, but fixes the issues
when multiple new shader related files are added, at once or out of order, to a game project
or Gem.
Overall the new ShaderAssetBuilder::CreateJobs() is around 40% faster
and, of course, handles source dependencies in a robust way.
* Added new test suite to AutomatedTesting project:
Gem/PythonTests/atom_renderer/test_Atom_ShaderBuildPipelineSuite.py
Bug fix to Gems/Atom/Asset/Shader/Code/Source/Editor/ShaderAssetBuilder.cpp
discovered thanks to the automated test suite. The idea is that
CreateJobs doesn't fail if the AZSL file doesn't exist. The failure is
deferred during ProcessJob. This way if the AZSL file exists the .shader
file is rebuilt automatically.
* For testability purposes and avoid memory leakage errors
during Unit Tests created the class ShaderBuilderUtility::IncludedFilesParser
Now accepts "# include <file>" with space between '#' and 'include'.
Also now accepts the '-' character inside the file path.
Added Unit Test to validate all cases of "#include <file>" parsing.
* Fixed linux runtime issues for Unit Tests in Atom_Asset_Shader.Tests
Signed-off-by: garrieta <garrieta@amazon.com>
* Fixed PathView `MakeRelativeTo` and `Append` functions path segment
comparisons when using the Windows path separator of '\'
The PathSegment comparisons were case-sensitive in both those functions
and now use `Internal::ComparePathSegments` function to perform the
appropriate case comparison based on the path separator value of the
Path class
Reverted the LocalFileIO::CheckInvalidWrite function back to not
lowercasing the assets alias and input path before invoking
`PathView::IsRelativeTo`
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Simplified the LocalFileIO::ConvertToAliasBuffer logic
Fix for the ArchiveTest
`IResourceList_Add_AbsolutePath_RemovesAndReplacesWithAlias` and
`TestArchiveViaFileIO` test
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added a PathIterable structure stores a non-heap container of normalized path segments of an input path.
Moved the PathParser logic to a PathParser.inl file
Removed dependency of the PathView::IsRelativeTo logic on FixedMaxPath
There is no longer a 1024 character limit when determining if a path is relative to a base
Added a GetNormalPathParts and AppendNormalPathParts to function and removed LexicallyNormalInplace to share the logic for creating a normalized path between IsRelativeTo and LexicallyNormal
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Clang PathIterable.inl build fix
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed Normalize and Relative Path functions initialize the result paths
With the correct path separator for the paths being transformed
Ported over the Custom Path Root Separator logic to the PathParser.inl
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated he Shader Preprocessor include path gather.
It now uses AZ::IO::Path for the path operations and checks if the path
exist before adding it to the list of include paths.
Finally the set logic has been removed for a simpler find_if check to
see if the include path already since in the project include paths
This fixes the Asset Processing issues with shader includes due to the
Path.inl changes
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed tail recursion call to AppendNormalPathParts to supply a PathView with the same path separator as the parent call
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Adding reference qualifier overloads to the Path class Native function
Removed the conversion operators from the Path class for converting to a
string_type&/const string_type&
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Release build fix for Windows
* Release build fix for Android
* Release build fix for Windows
* Release build fix for Android
* Release build fix for Linux
* Release build fix for Mac
* Release build fix for iOS
Signed-off-by: Steve Pham <spham@amazon.com>
Merge pull request #3574 from aws-lumberyard-dev/Atom/santorac/MoreShaderHotReloadFixes
Fixed a shader hot reload issue where the new root shader variant asset was not getting saved in the Shader object during OnAssetReloaded, it was only saved during OnAssetReady.
Fixed shader variant hot reload which was failing due to mismatched timestamps. The ShaderAsset was using microseconds and the ShaderVariantAsset was using system ticks. Since ticks will always be higher than microseconds, stale variants were not prevented from being used.
AtomSampleViewer's MaterialHotReloadTest script is passing again.
* Add linux equivalents for all the places that enumerate platforms
Signed-off-by: Chris Burel <burelc@amazon.com>
* Fix the AssetFileInfoListComparisonTest fixture to not hardcode the PC platform
Signed-off-by: Chris Burel <burelc@amazon.com>
* [ATOM-13679] ShaderAssetBuilder: Create UnitTest To Validate
STDOUT Data Capture From MCPP
azvsnprintf was being used improperly, in particular
in windows if the data to print was larger than the
local buffer in the stack, then azvsnprintf returns -1.
Also azvsnprintf needs a +1 in buffer size to accomodate for
the '\0' character at the end and that was not done.
Added UnitTest to validate all cases:
1. Data to print is smaller than the local buffer.
2. Data to print is the same size as the local buffer.
3. Data to print is bigger than the local buffer.
Signed-off-by: garrieta <garrieta@amazon.com>
* Fix for MacOS & Linux,
they require va_start to be called each time azvsnprintf is called
Signed-off-by: garrieta <garrieta@amazon.com>
* Enable Shader Compilation support for Linux
* Add azslc package 3P definition for download
* Add AzslcGeader.azsli nad PlatformHeader.hlsli for Linux/Null and Linux/Vulkan (Copy of Android/Vulkan)
* Enable trait PAL_TRAIT_BUILD_ATOM_ASSET_SHADER_SUPPORTED for Linux
* Add platform builder file dependencies
* Updated AZ_TRAIT_ATOM_SHADERBUILDER_AZSLC to use new 3P linux version of azslc
* Add AtomShaderCapabilities.json, PlatformLimits.azasset for Linux (Copy from Android)
* Fix paths/values for Vulkan_Traits_Linux.h
* Enable trait PAL_TRAIT_BUILD_ATOM_ASSET_SHADER_SUPPORTED for Linux
* Enabled AZ_TRAIT_ATOM_VULKAN_LAYER_LUNARG_STD_VALIDATION_SUPPORT for Linux/Vulkan
* Added AzslcHeader.azsli, PlatformHeader.hlsli dependencies for Linux/Null and Linux/Vulkan in the platform builders for windows and mac for consistency
Signed-off-by: spham-amzn <spham@amazon.com>
* Final update copyright headers to reference license files at the repo root
Signed-off-by: spham <spham@amazon.com>
* Fix copyright validator unit tests to support the stale O3DE header scenario
Signed-off-by: spham <spham@amazon.com>
ATOM-15837 Remove Unused Shader System Related *2 Classes From Stabilization Branch.
Testing: ASV full test suite on dx12 and vulkan. Only saw known issues.
Cherrypicked b7f6b57e16 from stabilization branch because I knew it would have conflicts I needed to resolve.
See https://github.com/aws-lumberyard/o3de/pull/1462
This removes mostly benign (but noisy) messages about "Failed to find builder dependency".
ATOM-15136 Builder dependency errors reported in mainline
ATOM-15134 Replace GetPossibleDepenencyPaths Approach with Source Dependencies
SrgLayoutBuilder.cpp conflicted because it had been removed on development. The changes in this file are no longer relevant.
ShaderVariantAssetBuilder.cpp conflicted only because there were formatting changes inside the AddShaderAssetJobDependency() function.
This removes mostly benign (but noisy) messages about "Failed to find builder dependency".
Changed AssetUtils::GetPossibleDepenencyPaths to return all possible source paths, rather than stopping when one is found. This function is now used to report a list of all possible source dependencies, so that CreateJobs will get called by the AP whenever a file shows up at one of those locations. If a file was missing before and then appears, this will cause the builders to wake up and add the appropriate job dependencies on the new files. In ShaderVariantAssetBuilder and MaterialBuilder, we now use GetPossibleDepenencyPaths to report source dependencies rather than job dependencies. We only report a job dependency when the actual source file has been identified. This should all now be consistent with the intended design of the AP's dependency systems (the prior approach was a hack based on misunderstanding of what source dependencies are).
SrgLayoutBuilder's change is a bit tricky. The above changes did not fix all of the "Failed to find builder dependency" messages because AzslBuilder sometimes skips particular files in CreateJobs. When this happens, it is invalid to report an AzslBuilder job dependency on that file. So I copied the same conditional code that is used to skip an azsli file in CreateJobs, and used that to skip AddAzslBuilderJobDependency() in SrgLayoutBuilder as well.
With all these changes combined, I do think we've solved the issue where jobs fail to evict outdated jobs, as described in ATOM-15134. However, we are not yet seeing the iteration time improvements we were hoping for. Before these changes I was seeing roughly a 0.5 minute delay for the initial change, and a 2 minute delay for a subsequent change. With these changes it's more like 0.5 mibutes and 1.5 minutes. It appears that the AP scan is being starved by all the AssetBuilder processing going on, and perhaps IO contention. I suspect that this will be greatly improved on the development branch where we no longer have AzslBuilder and SrgLayoutBuilder slowing things down.
ATOM-15136 Builder dependency errors reported in mainline
ATOM-15134 Replace GetPossibleDepenencyPaths Approach with Source Dependencies
* [ATOM-15472] Shader Build Pipeline: Remove Deprecated Files And Functions That
Predate The Shader Supervariants
These are the essential impactful changes as a result of deprecating the
ShaderResourceGroupAsset.
* Addressed feedback by @moudgils. Better comments in header files.
* More updates related with deprecation of ShaderResourceGroupAsset
* Deleted the temporary version 2 classes.
* Updated version of the shader asset builders.
* Updated version of all the shader related classes impacted
by the Supervariant concept and deprecation of ShaderResourceGroupAsset
* Changes to *.pass and DGI, Reflections and RayTracing.
* changes to material related assets
* changes to core lights
* Changes to auxgeom/dynamic draw.
* changes to decals, lyshine, imguipass
* changes to RPI Pass classes
* Shader for SceneSrg, ViewSrg and ForwardPass Srgs.
* changes to mesh, skinned mesh, Morphtarget.
* Fixes to RayTracingPass.cpp & now allow empty srg in shaders.
* Updated Atom_RPI.Tests
* Simplified InstanceDatabase by removing AddHandler
------------------------------------------------------------------------------------
* Updated DiffuseGI precompiled shaders.
Added RayTracingSceneSrg and RayTracingMaterialSrg shader asset.
Updated ShaderAssetCreator::Clone to handle the supervariant when processing root variants.
Co-authored-by: Doug McDiarmid <dmcdiar@amazon.com>
------------------------------------------------------------------------------------
* Changed semantics for some PassSrg to SRG_PerPass_WithFallback.
AuxGeom/FixedShapeProcessor.cpp requires SRG_PerDraw on ObjectSrg.
Removed names of SceneSrg and ViewSrg from RPISystemDescriptor.cpp
* Moved ShaderLib/Atom/Features/DummyEntryFunctions.azsli
To Gems/Atom/RPI/Assets/ShaderLib/Atom/RPI/DummyEntryFunctions.azsli
Removed redundant checking for finalization in
ShaderResourceGroupLayout.cpp
* Fixed race condition bug for Shader::FindOrCreate.
InstanceDatabase<>::CreateInstance() needs to be atomic
for instance creation and initialization.
Added optional InstanceHandler::CreateFunctionWithParams to accomodate
to the needs of Instances that need more than an asset reference
to be able to be created an initialzed.
Removed ShaderResourceGroup::FindOrCreate() only ::Create is available
now.
* Renamed scene_and_view_srgs.* as SceneAndViewSrgs.*
Changed GetAzslFileOfOrigin for GetUniqueId
* Fixed unit tests.
* Reverted the serialization name of m_uniqueId back to
"m_azslFileOfOrigin" so precompiled shaders don't fail
in layout comparison.
* Fixed AtomCore.Tests
Removed non-applicable test. InstanceDatabase.AddHandler() is not
available anymore.
* The Null rhi is re-enabled for shader compilation.
Signed-off-by: garrieta <garrieta@amazon.com>
* [ATOM-15618] Shader Build Pipeline: Add UnitTest To Validate Shader Compiler
Argument Processing Introduced With The New Supervariant System
- Added new test suite in
Gems/Atom/Asset/Shader/Code/Tests/SupervariantCmdArgumentTests.cpp
- Refactored and improved the previously existing classes:
GlobalBuildOptions, PreprocessorOptions and ShaderCompilerArguments
to work well with the new ShaderSourceData::SupervariantInfo.
- Moved command line argument processing function out of
ShaderCompilerArguments and into its own utility namespace in
Atom/RHI.Edit/Utils.h
Signed-off-by: garrieta <garrieta@amazon.com>
santorac
galibzon
lumberyard-employee-dm
Esteban Papp
Steve Pham
srikappa-amzn
srikappa-amzn
Chris Burel
chcurran
pappeste
antonmic
Chris Galvan
Gene Walters
Chris Santora
Terry Michaels