Merge pull request #4338 from aws-lumberyard-dev/Atom/rbarrand/MaterialVersionUpdate
## Overview
Added support for automatically updating .material files when the property layout of the .materialtype changes. At this stage, we only support a "rename" operation for renaming or moving material properties. We can add more operations in the future if needed (this will require some improvement to how we represent the data internally, but we can deal with that as the need arises).
It's important to note that we currently have a hybrid dependency model for the material system, where materials can depend on materialtypes as job dependencies (property names are processed at asset build time), or material property names can be stored in the cooked assets (property names are processed and resolved at runtime when loading material assets). This means there are two places where we need to apply material property rename auto-updates: in the tools and in the runtime. So you will find that we have ApplyVersionUpdates() functions in both MaterialSourceData and MaterialAsset. (I hope we can move away from this hybrid approach at some point so this can all be simplified, but that depends on new Asset Processor features we don't have yet).
## Main changes
- Added version and versionUpdates structures for .materialtype files, MaterialTypeSourceData, MaterialTypeAssetCreator, etc.
- The .materialtype version number is now at the top level instead of inside the propertyLayout section, because in the future there are other ways the material type could change besides the property layout which might require version auto-updates. (The AP will fail if the version is found in the old location, and tell the user where it should be moved).
- Added ApplyVersionUpdates() and ApplyPropertyRenames() utility functions to facilitate the auto-updates.
- Updated MaterialTypeSourceData::FindProperty(name) to support renames because it must find the property data while loading .material files in order to resolve property data types.
- These new functions will report warnings when they detect that source files are out of date, and recommend the user update them. The easiest way to do this is open it in the Material Editor and save. (which can be scripted in python if necessary)
- Renamed the .material file format "propertyLayoutVersion" to "materialTypeVersion" which is more accurate. This shouldn't hurt existing data as this field wasn't actually used for anything before.
## Unit test improvements
- The new code is well unit tested.
- MaterialSourceDataTests
- Updated to include both a .materialtype file and a MaterialTypeAsset for the test material type. Both are used by the MaterialTypeSourceData class.
- The default test material type now includes some version update steps; these are only used for version update tests and won't impact the other test functions.
- Updated the path for storing temp files to disk, to just be in a "temp" folder in the exe path. (Originally they were saved to the gem folder near MaterialSourceDataTests.cpp, but at some point someone changed it to be under the exe folder, so there's no reason to use the full gem path anymore).
## Other changes
- Fixed AzCore Utils WriteFile() to create the file if it doesn't exist already.
- Updated all .materialtype files to have the version number in the new position.
- Fixed AssetSystemStub to normalize asset paths for more reliable lookup in unit tests.
* Fixed warnings of unused marks, renamed ctest_pytest.ini to pytest.ini to better consistency on runs
* Fixed some test suites to run propertly
* Fix missing arguments
* Fixed missing cmakelists and renamed missing file
* Temp disable editor_testing_tests as timeout in jenkins
* Remove the definition of a 'Stub' Wwise Gem
This removes the conditional compilation of a '.Stub' version of the
Wwise Gem. Now, all we do is not define any of the AudioEngineWwise
targets in CMake whenever Wwise is not available (SDK not found or
platform unsupported).
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Remove the remaining stub files, no longer needed
This removes the files.cmake and stub module cpp files that made up the
.Stub version of this Gem. These are no longer needed.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
- Add a new automated test PrefabBasicWorkflow_CreateReparentAndDetachPrefab for verifying prefab detachment basic workflow.
- Add a new automated test PrefabBasicWorkflow_CreateAndDuplicatePrefab for verifying prefab detachment basic workflow.
- Fix a bug related to sets of entity ids in Reparent helper function .
- This rendering technique reduces the memory required per pipeline by 750MB compared to the PPLL technique!!
- Using 3 screen buffers representing the closest 3 hair gragments depths.
- Going through second geometry pass that disqualify any fragment further than the third depth, the technique blends the three closes shaders hair fragments.
- Supports the Marschner lighting model (big change from the original TressFX 4.1 implementation)
- The current technique is almost twice the performance of the PPLL but not quite as advacned when come to final visual quality.
Remarks:
Unlike the PPLL, this technique is still lacking some of the advanced features added to the PPLL such as
1. Back lobe (TT) conseal by depth comparison
2. Thickness dependency in light transfer (mainly TT)
3. Allowing TT transfer for thin separated hair strands (might be supported by default with no distinction)
Signed-off-by: Adi-Amazon <Adi Bar-Lev 82479970+Adi-Amazon@users.noreply.github.com>
Signed-off-by: Adi-Amazon <Adi Bar-Lev barlev@amazon.com>
Signed-off-by: Adi-Amazon <Adi Bar-Lev 82479970+Adi-Amazon@users.noreply.github.com>
Signed-off-by: Adi-Amazon <Adi Bar-Lev barlev@amazon.com>
Co-authored-by: Adi-Amazon <Adi Bar-Lev 82479970+Adi-Amazon@users.noreply.github.com>
(It's a simple enough change to make manually, and making .materialtype is an uncommon workflow, so not worth doing this automatically).
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
* Skipping GraphUpdate test due to LC node issue in Debug configuration
Signed-off-by: jckand-amzn <jckand@amazon.com>
* Reverting test type for GraphUpdates test to EditorSharedTest
Signed-off-by: jckand-amzn <jckand@amazon.com>
* Updating xfail reason for GraphUpdates test with GitHub issue link
Signed-off-by: jckand-amzn <jckand@amazon.com>
* Initial support for importing Json files within other Json files
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add some test cases for testing/iterating on the Json import work. Fix MacOS AzTestRunner module loading bug.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* The import resolver can take the allocator as a parameter to Load/StoreImports() instead of storing a copy.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Fix assert
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Some rework of the JsonImport feature. Base test cases pass. More complex test cases need to be added.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Add test case for testing nested imports.
2. Initialize rapidjson value to fix assert.
3. Fix bug found in merge patch creation.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Update the Resolver class member functions to return proper result codes.
2. Add the wrapper functions for resolving/restoring imports to the JsonSerialization class.
3. Add new test case.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add test cases for import + patches. Fix bug found when patching import. Rename test cases.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Add ApplyPatch() function to BaseJsonImporter.
2. Move patch logic out of ResolveImport() and into ApplyPatch()
3. Get rid of the custom RestoreImport implementation in the tests since it was the same as the base version.
4. Add test case for patching nested imports.
5. Update merge patch outcome reporting logic to work for nested object patches.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Add a CreatePatch() function to BaseJsonImporter to match the ApplyPatch() function.
2. Reorganize some responsibilities between RestoreImports(), RestoreImport() and CreatePatch() to make ResolveImports() and RestoreImports() more symmetrical.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Combine result code in code path where we add empty object to path
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add test case for inserting a new import into an existing object.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Use == instead of Compare() for comparing file paths.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Address some PR feedback.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Address additional PR feedback
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add missing includes to fix non-unity build
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Fix build error. Address additional feedback.
Signed-off-by: amzn-sj <srikkant@amazon.com>
`LaunchProcess()` on Linux works by calling `fork` then `execvpe`. `fork`
is used to copy a running process, generating a new child process. The new
child starts running from the location where the parent was running, from
whatever thread from the parent called `fork`. The child process only gets
one thread, however. If a different thread in the parent process had locked
a mutex, that mutex is also locked in the child process. Since that
separate thread is not present in the child, the mutex remains locked in
the child, with no way to unlock it. So it is important that as little work
as possible happens between the call to `fork` and to `execvpe`.
Previously, this code was trying to report an error that may have occurred
from calling `execvpe`. It was doing that by calling `AZ_TracePrintf`. That
function does lots of things, including trying to make an EBus call, which
looks up a variable in the `AZ::Environment` instance, which has a global
mutex. If there was some other thread that had that mutex locked when the
`fork` call was made, the subprocess would deadlock, and the parent process
would also deadlock waiting for the child to finish.
This solves that issue by removing the call to `AZ_TracePrintf` from the
subprocess code path. Instead, the parent process sets up a pipe for the
child process to write to in case the call to `execvpe` fails (the
self-pipe trick). The parent then reads from that pipe. If it reads no
data, `execvpe` worked and there's no error. If it does read data, the data
to be read is the errno from the failed `execvpe` call made by the child.
The parent can then use `strerror()` to report the error.
Fixes#4702.
Signed-off-by: Chris Burel <burelc@amazon.com>
* Fixed create-project/create-gem to use relative paths directly
It was previously making relative paths, relative to the default_projects_folder and default_gems_folders
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed uppercase of "Templates" param in get_registered call.
That method looks up the restricted template directory location based on the lowercase key of "templates"
Updated argparse doc to fix typos or clarify which directories a path is relative to.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
Also improved the MaterialAssetTests UpgradeMaterialAsset() to focus on testing the inputs and outputs of the class rather than the private internal data.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
Also rearranged some logic to simplify the code that loops over rename actions, avoiding making unnecessary additional maps.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
- Added MaterialSourceData::ApplyVersionUpdates() for updating the properties. This should be called by tools after loading the MaterialSourceData. (But can be omitted if a tool wants to read the data exactly as it appears in the .material file).
- Updated MaterialTypeSourceData::FindProperty to support applying version update renames, including a ApplyPropertyRenames utility function, which are necessary for MaterialSourceData to be able to find the necessary property definitons while loading.
- Added a new context struct to JsonMaterialPropertyValueSerializer for passing down the material type version number, to help with applying property renames.
- Renamed the .material file format "propertyLayoutVersion" to "materialTypeVersion" which is more accurate. This shouldn't hurt existing data as this field wasn't actually used for anything before.
- Updated Material Editor to again store the material type version number in .material files.
MaterialSourceDataTests updates...
- Updated to include both a .materialtype file and a MaterialTypeAsset for the test material type. Both are used by the MaterialTypeSourceData class.
- The default test material type now includes some version update steps; these are only used for version update tests and won't impact the other test functions.
- Updated the path for storing temp files to disk, to just be in a "temp" folder in the exe path. (Originally they were saved to the gem folder near MaterialSourceDataTests.cpp, but at some point someone changed it to be under the exe folder, so there's no reason to use the full gem path anymore).
MaterialTypeSourceDataTests updates...
- Moved some code that was accidentally added to LoadAllFieldsUsingOldFormat but should have been in LoadAndStoreJson_AllFields.
- Added test cases for unsupported version update operations
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
Benjamin Jillich
santorac
Chris Galvan
Jonny Galloway
AMZN-Phil
jromnoa
Nicholas Van Sickle
AMZN-AlexOteiza
Danilo Aimini
amzn-phist
AMZN-Phil
jromnoa
Steve Pham
chiyenteng
hershey5045
srikappa-amzn
Adi Bar-Lev
Scott Romero
jckand-amzn
SJ
Chris Burel
lumberyard-employee-dm
Michael Pollind
Guthrie Adams