The main addition here is the MaterialNameContext class which represents the concept of a namespace for properties, shader options, and SRG fields. This concept was already somewhat supported in LuaMaterialFunctor through bespoke "prefix" fields, but I have generalized it be available for all material functors. Note that I have not yet updated the other material functor types to ensure they take advantage of this feature, that will be in another commit.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
I also noticed that JsonFileLoadContext was no longer used (see https://github.com/o3de/o3de/pull/7010) so I was able to remove all that code.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
In some cases it may be acceptable to do non-const things in a const function as long as it is only manipulating internal data, and the public facing API returns the same values as before. But in this case, the IsFinalized function is a public facing API that would have a different result after GetPropertyValues was called.
I also updated a couple other minor things from code review feedback.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
I made MaterialAsset::Finalize private so I could add some parameters specifically for MaterialAssetCreator to use. Now MaterialAssetCreator::Begin has an option to finalize the material or not.
Moved MaterialAssetCreatorCommon::ValidateDataType to MaterialPropertyDescriptor as "ValidateMaterialPropertyDataType" so that MaterialAsset::Finalize could use it too
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
Before, the material builder was loading the MaterialTypeAsset and doing some processing with it, but was avoiding declaring job dependencies that would cause reprocessing lots of assets when a shader or .materialtype file changes. Reading the asset data isn't safe when not declaring a job dependency (or when declaring a weak job dependency like OrderOnce which is the case here). This caused to several known bugs.
The main change here is it no longer loads the MaterialTypeAsset at all; all other changes flow from there.
The biggest changes (when deferred material processing is enabled) are ...
1) MaterialSourceData no longer loads MaterialTypeAsset. All it really needs is to determine whether a string is an image file reference or an enum value, which is easy to do by just looking for the "." for the extension.
2) MaterialAssetCreator no longer produces a finalized material asset. It no longer uses MaterialAssetCreatorCommon because that only produces a non-finalized MaterialAsset, which has very different needs for the SetPropertyValue function. (We could consider merging MaterialAssetCreatorCommon into MaterialTypeAssetCreator since that's the only subclass at this point). And it doesn't do any validation against the properties layout since that can be done at runtime.
3) Moved processing of enum property values from MaterialSourceData to MaterialAsset::Finalize (this was the only thing being done in the builder that actually needed to read the material type asset data).
Also...
- Updated the MaterialAsset class mostly to clarify and formalize the two different modes it can be in: whether it is finalized or not.
- Merged the separate "IncludeMaterialPropertyNames" registry settings from MaterialConverterSystemComponent and MaterialBuilder into one "FinalizeMaterialAssets" setting used for both.
- Removed MaterialSourceData::ApplyVersionUpdates. Now the flow of data is the same regardless of whether the materials are finalized by the AP or at runtime. Version updates are always applied on the MaterialAsset.
- Added a validation check to MaterialTypeAssetCreator ensuring that once a property is renamed, the old name can never be used again for a new property. This assumption was already made previously, but not formalized, in that Material::FindPropertyIndex does not expect every caller to provide a version number for the material property name, also the material asset's list of raw property names was never versioned. The only way for this to be a safe assumption is to prevent reuse of old names.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
The main reason for this is to give consistent results between the AP and Material Editor, where a placholder texture can be used if a texture is missing. Otherwise, you could get a placeholder texture in Material Editor and stale data in the runtime; this inconsistency would be confusing.
As a consequence, it is possible for example that the user could mess up the name of a property in a .material file and not notice the problem because it is now a warning instead of an error. If warnings-as-errors is desirable, you can enable the new "/O3DE/Atom/RPI/MaterialBuilder/WarningsAsErrors" registry setting.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
(It's a simple enough change to make manually, and making .materialtype is an uncommon workflow, so not worth doing this automatically).
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
- Added MaterialSourceData::ApplyVersionUpdates() for updating the properties. This should be called by tools after loading the MaterialSourceData. (But can be omitted if a tool wants to read the data exactly as it appears in the .material file).
- Updated MaterialTypeSourceData::FindProperty to support applying version update renames, including a ApplyPropertyRenames utility function, which are necessary for MaterialSourceData to be able to find the necessary property definitons while loading.
- Added a new context struct to JsonMaterialPropertyValueSerializer for passing down the material type version number, to help with applying property renames.
- Renamed the .material file format "propertyLayoutVersion" to "materialTypeVersion" which is more accurate. This shouldn't hurt existing data as this field wasn't actually used for anything before.
- Updated Material Editor to again store the material type version number in .material files.
MaterialSourceDataTests updates...
- Updated to include both a .materialtype file and a MaterialTypeAsset for the test material type. Both are used by the MaterialTypeSourceData class.
- The default test material type now includes some version update steps; these are only used for version update tests and won't impact the other test functions.
- Updated the path for storing temp files to disk, to just be in a "temp" folder in the exe path. (Originally they were saved to the gem folder near MaterialSourceDataTests.cpp, but at some point someone changed it to be under the exe folder, so there's no reason to use the full gem path anymore).
MaterialTypeSourceDataTests updates...
- Moved some code that was accidentally added to LoadAllFieldsUsingOldFormat but should have been in LoadAndStoreJson_AllFields.
- Added test cases for unsupported version update operations
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
This revealed that the approach of reflecting both the old "id" and the new "name" would not work, because whenn saving it would write out both fields. So I decied to just give up on backward compatibility. This will be much cleaner than trying to continue supporting "id" as a field name, it is uncommon for users to make their own material types at this point, and if they have made some it is very easy to search and replace "id" with "name" update their files.
All .materialtype files have been updated. RPI unit tests now pass. ASV still passes.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
* Final update copyright headers to reference license files at the repo root
Signed-off-by: spham <spham@amazon.com>
* Fix copyright validator unit tests to support the stale O3DE header scenario
Signed-off-by: spham <spham@amazon.com>
Cherrypicked b7f6b57e16 from stabilization branch because I knew it would have conflicts I needed to resolve.
See https://github.com/aws-lumberyard/o3de/pull/1462
This removes mostly benign (but noisy) messages about "Failed to find builder dependency".
ATOM-15136 Builder dependency errors reported in mainline
ATOM-15134 Replace GetPossibleDepenencyPaths Approach with Source Dependencies
SrgLayoutBuilder.cpp conflicted because it had been removed on development. The changes in this file are no longer relevant.
ShaderVariantAssetBuilder.cpp conflicted only because there were formatting changes inside the AddShaderAssetJobDependency() function.
This removes mostly benign (but noisy) messages about "Failed to find builder dependency".
Changed AssetUtils::GetPossibleDepenencyPaths to return all possible source paths, rather than stopping when one is found. This function is now used to report a list of all possible source dependencies, so that CreateJobs will get called by the AP whenever a file shows up at one of those locations. If a file was missing before and then appears, this will cause the builders to wake up and add the appropriate job dependencies on the new files. In ShaderVariantAssetBuilder and MaterialBuilder, we now use GetPossibleDepenencyPaths to report source dependencies rather than job dependencies. We only report a job dependency when the actual source file has been identified. This should all now be consistent with the intended design of the AP's dependency systems (the prior approach was a hack based on misunderstanding of what source dependencies are).
SrgLayoutBuilder's change is a bit tricky. The above changes did not fix all of the "Failed to find builder dependency" messages because AzslBuilder sometimes skips particular files in CreateJobs. When this happens, it is invalid to report an AzslBuilder job dependency on that file. So I copied the same conditional code that is used to skip an azsli file in CreateJobs, and used that to skip AddAzslBuilderJobDependency() in SrgLayoutBuilder as well.
With all these changes combined, I do think we've solved the issue where jobs fail to evict outdated jobs, as described in ATOM-15134. However, we are not yet seeing the iteration time improvements we were hoping for. Before these changes I was seeing roughly a 0.5 minute delay for the initial change, and a 2 minute delay for a subsequent change. With these changes it's more like 0.5 mibutes and 1.5 minutes. It appears that the AP scan is being starved by all the AssetBuilder processing going on, and perhaps IO contention. I suspect that this will be greatly improved on the development branch where we no longer have AzslBuilder and SrgLayoutBuilder slowing things down.
ATOM-15136 Builder dependency errors reported in mainline
ATOM-15134 Replace GetPossibleDepenencyPaths Approach with Source Dependencies
* [ATOM-15472] Shader Build Pipeline: Remove Deprecated Files And Functions That
Predate The Shader Supervariants
These are the essential impactful changes as a result of deprecating the
ShaderResourceGroupAsset.
* Addressed feedback by @moudgils. Better comments in header files.
* More updates related with deprecation of ShaderResourceGroupAsset
* Deleted the temporary version 2 classes.
* Updated version of the shader asset builders.
* Updated version of all the shader related classes impacted
by the Supervariant concept and deprecation of ShaderResourceGroupAsset
* Changes to *.pass and DGI, Reflections and RayTracing.
* changes to material related assets
* changes to core lights
* Changes to auxgeom/dynamic draw.
* changes to decals, lyshine, imguipass
* changes to RPI Pass classes
* Shader for SceneSrg, ViewSrg and ForwardPass Srgs.
* changes to mesh, skinned mesh, Morphtarget.
* Fixes to RayTracingPass.cpp & now allow empty srg in shaders.
* Updated Atom_RPI.Tests
* Simplified InstanceDatabase by removing AddHandler
------------------------------------------------------------------------------------
* Updated DiffuseGI precompiled shaders.
Added RayTracingSceneSrg and RayTracingMaterialSrg shader asset.
Updated ShaderAssetCreator::Clone to handle the supervariant when processing root variants.
Co-authored-by: Doug McDiarmid <dmcdiar@amazon.com>
------------------------------------------------------------------------------------
* Changed semantics for some PassSrg to SRG_PerPass_WithFallback.
AuxGeom/FixedShapeProcessor.cpp requires SRG_PerDraw on ObjectSrg.
Removed names of SceneSrg and ViewSrg from RPISystemDescriptor.cpp
* Moved ShaderLib/Atom/Features/DummyEntryFunctions.azsli
To Gems/Atom/RPI/Assets/ShaderLib/Atom/RPI/DummyEntryFunctions.azsli
Removed redundant checking for finalization in
ShaderResourceGroupLayout.cpp
* Fixed race condition bug for Shader::FindOrCreate.
InstanceDatabase<>::CreateInstance() needs to be atomic
for instance creation and initialization.
Added optional InstanceHandler::CreateFunctionWithParams to accomodate
to the needs of Instances that need more than an asset reference
to be able to be created an initialzed.
Removed ShaderResourceGroup::FindOrCreate() only ::Create is available
now.
* Renamed scene_and_view_srgs.* as SceneAndViewSrgs.*
Changed GetAzslFileOfOrigin for GetUniqueId
* Fixed unit tests.
* Reverted the serialization name of m_uniqueId back to
"m_azslFileOfOrigin" so precompiled shaders don't fail
in layout comparison.
* Fixed AtomCore.Tests
Removed non-applicable test. InstanceDatabase.AddHandler() is not
available anymore.
* The Null rhi is re-enabled for shader compilation.
Signed-off-by: garrieta <garrieta@amazon.com>
santorac
Robin
srikappa-amzn
srikappa-amzn
Steve Pham
chcurran
Chris Galvan
Gene Walters
Chris Santora
galibzon
alexpete