Merge pull request #4338 from aws-lumberyard-dev/Atom/rbarrand/MaterialVersionUpdate
## Overview
Added support for automatically updating .material files when the property layout of the .materialtype changes. At this stage, we only support a "rename" operation for renaming or moving material properties. We can add more operations in the future if needed (this will require some improvement to how we represent the data internally, but we can deal with that as the need arises).
It's important to note that we currently have a hybrid dependency model for the material system, where materials can depend on materialtypes as job dependencies (property names are processed at asset build time), or material property names can be stored in the cooked assets (property names are processed and resolved at runtime when loading material assets). This means there are two places where we need to apply material property rename auto-updates: in the tools and in the runtime. So you will find that we have ApplyVersionUpdates() functions in both MaterialSourceData and MaterialAsset. (I hope we can move away from this hybrid approach at some point so this can all be simplified, but that depends on new Asset Processor features we don't have yet).
## Main changes
- Added version and versionUpdates structures for .materialtype files, MaterialTypeSourceData, MaterialTypeAssetCreator, etc.
- The .materialtype version number is now at the top level instead of inside the propertyLayout section, because in the future there are other ways the material type could change besides the property layout which might require version auto-updates. (The AP will fail if the version is found in the old location, and tell the user where it should be moved).
- Added ApplyVersionUpdates() and ApplyPropertyRenames() utility functions to facilitate the auto-updates.
- Updated MaterialTypeSourceData::FindProperty(name) to support renames because it must find the property data while loading .material files in order to resolve property data types.
- These new functions will report warnings when they detect that source files are out of date, and recommend the user update them. The easiest way to do this is open it in the Material Editor and save. (which can be scripted in python if necessary)
- Renamed the .material file format "propertyLayoutVersion" to "materialTypeVersion" which is more accurate. This shouldn't hurt existing data as this field wasn't actually used for anything before.
## Unit test improvements
- The new code is well unit tested.
- MaterialSourceDataTests
- Updated to include both a .materialtype file and a MaterialTypeAsset for the test material type. Both are used by the MaterialTypeSourceData class.
- The default test material type now includes some version update steps; these are only used for version update tests and won't impact the other test functions.
- Updated the path for storing temp files to disk, to just be in a "temp" folder in the exe path. (Originally they were saved to the gem folder near MaterialSourceDataTests.cpp, but at some point someone changed it to be under the exe folder, so there's no reason to use the full gem path anymore).
## Other changes
- Fixed AzCore Utils WriteFile() to create the file if it doesn't exist already.
- Updated all .materialtype files to have the version number in the new position.
- Fixed AssetSystemStub to normalize asset paths for more reliable lookup in unit tests.
* Introduce Outliner button to simplify Prefab editing
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Fix Focus Mode and disabled entities colors
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Fix - propagate the event if the OnOutlinerItemClick function returns false.
This does not change current behavior but makes more sense in the context of future handlers.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Adjust disabled colors to match UX recommendations.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Display the edit button even if the prefab is disabled. Remove prefabWip check (it will be removed for focus mode by the time this goes in). Default to disabled capsule color for borders to save on checks.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Disable edit button on disabled prefabs as it caused conflicts in nested prefabs. May explore that possibility later.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Change disabled text color to be darker, as asked by UX.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
- Add a new automated test PrefabBasicWorkflow_CreateReparentAndDetachPrefab for verifying prefab detachment basic workflow.
- Add a new automated test PrefabBasicWorkflow_CreateAndDuplicatePrefab for verifying prefab detachment basic workflow.
- Fix a bug related to sets of entity ids in Reparent helper function .
* Initial support for importing Json files within other Json files
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add some test cases for testing/iterating on the Json import work. Fix MacOS AzTestRunner module loading bug.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* The import resolver can take the allocator as a parameter to Load/StoreImports() instead of storing a copy.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Fix assert
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Some rework of the JsonImport feature. Base test cases pass. More complex test cases need to be added.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Add test case for testing nested imports.
2. Initialize rapidjson value to fix assert.
3. Fix bug found in merge patch creation.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Update the Resolver class member functions to return proper result codes.
2. Add the wrapper functions for resolving/restoring imports to the JsonSerialization class.
3. Add new test case.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add test cases for import + patches. Fix bug found when patching import. Rename test cases.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Add ApplyPatch() function to BaseJsonImporter.
2. Move patch logic out of ResolveImport() and into ApplyPatch()
3. Get rid of the custom RestoreImport implementation in the tests since it was the same as the base version.
4. Add test case for patching nested imports.
5. Update merge patch outcome reporting logic to work for nested object patches.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. Add a CreatePatch() function to BaseJsonImporter to match the ApplyPatch() function.
2. Reorganize some responsibilities between RestoreImports(), RestoreImport() and CreatePatch() to make ResolveImports() and RestoreImports() more symmetrical.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Combine result code in code path where we add empty object to path
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add test case for inserting a new import into an existing object.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Use == instead of Compare() for comparing file paths.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Address some PR feedback.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Address additional PR feedback
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add missing includes to fix non-unity build
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Fix build error. Address additional feedback.
Signed-off-by: amzn-sj <srikkant@amazon.com>
`LaunchProcess()` on Linux works by calling `fork` then `execvpe`. `fork`
is used to copy a running process, generating a new child process. The new
child starts running from the location where the parent was running, from
whatever thread from the parent called `fork`. The child process only gets
one thread, however. If a different thread in the parent process had locked
a mutex, that mutex is also locked in the child process. Since that
separate thread is not present in the child, the mutex remains locked in
the child, with no way to unlock it. So it is important that as little work
as possible happens between the call to `fork` and to `execvpe`.
Previously, this code was trying to report an error that may have occurred
from calling `execvpe`. It was doing that by calling `AZ_TracePrintf`. That
function does lots of things, including trying to make an EBus call, which
looks up a variable in the `AZ::Environment` instance, which has a global
mutex. If there was some other thread that had that mutex locked when the
`fork` call was made, the subprocess would deadlock, and the parent process
would also deadlock waiting for the child to finish.
This solves that issue by removing the call to `AZ_TracePrintf` from the
subprocess code path. Instead, the parent process sets up a pipe for the
child process to write to in case the call to `execvpe` fails (the
self-pipe trick). The parent then reads from that pipe. If it reads no
data, `execvpe` worked and there's no error. If it does read data, the data
to be read is the errno from the failed `execvpe` call made by the child.
The parent can then use `strerror()` to report the error.
Fixes#4702.
Signed-off-by: Chris Burel <burelc@amazon.com>
- Added MaterialSourceData::ApplyVersionUpdates() for updating the properties. This should be called by tools after loading the MaterialSourceData. (But can be omitted if a tool wants to read the data exactly as it appears in the .material file).
- Updated MaterialTypeSourceData::FindProperty to support applying version update renames, including a ApplyPropertyRenames utility function, which are necessary for MaterialSourceData to be able to find the necessary property definitons while loading.
- Added a new context struct to JsonMaterialPropertyValueSerializer for passing down the material type version number, to help with applying property renames.
- Renamed the .material file format "propertyLayoutVersion" to "materialTypeVersion" which is more accurate. This shouldn't hurt existing data as this field wasn't actually used for anything before.
- Updated Material Editor to again store the material type version number in .material files.
MaterialSourceDataTests updates...
- Updated to include both a .materialtype file and a MaterialTypeAsset for the test material type. Both are used by the MaterialTypeSourceData class.
- The default test material type now includes some version update steps; these are only used for version update tests and won't impact the other test functions.
- Updated the path for storing temp files to disk, to just be in a "temp" folder in the exe path. (Originally they were saved to the gem folder near MaterialSourceDataTests.cpp, but at some point someone changed it to be under the exe folder, so there's no reason to use the full gem path anymore).
MaterialTypeSourceDataTests updates...
- Moved some code that was accidentally added to LoadAllFieldsUsingOldFormat but should have been in LoadAndStoreJson_AllFields.
- Added test cases for unsupported version update operations
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
- Prevent P4 thread to run if we cannot detect the P4 command to begin with
- Add a trait to disable calling the parent ComponentApplication::Destroy(), instead calling _exit() to skip the module unloading on exit
Signed-off-by: Steve Pham <spham@amazon.com>
* Initialize the PrefabFocusHandler on context reset, to also cover the case of a new level being created on the welcome screen.
Relax checks/restrictions on refreshes to cover cases where an instance is reused by the Prefab EOS.
Refresh the breadcrumbs when a container is renamed and when a change is propagated to the instances to ensure the correct names are displayed.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Rename m_isInitialized to m_initialized in PrefabFocusHandler
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Use find_if to detect when a container entity in the focus path has been renamed.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Renaming and commenting variables in PrefabFocusHandler.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Undo minor naming change
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
* Replace lazy initialization and have the UI side initialize the Editor calls in PrefabFocusHandler.
This should prevent issues with focus mode trying to access these interfaces in non-editor applications.
Signed-off-by: Danilo Aimini <82231674+AMZN-daimini@users.noreply.github.com>
Move the existing Qt Toast Notification QWidgets, EBuses and logic from the GraphCanvas gem into AzQtComponents and AzToolsFramework so they can be re-used.
Signed-off-by: AMZN-alexpete <26804013+AMZN-alexpete@users.noreply.github.com>
* {lyn7065} adding ProcPrefab Prefab::Tempate flag method
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
* updated based on comments
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
* moved validation logic to IsValid()
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
* added more guards around the source string
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
The test was relying on immediate updates from the prefab system which are now scheduled for a later tick - this reworks the tests to wait for deferred updates before validating state
Signed-off-by: nvsickle <nvsickle@amazon.com>
* Added anchor key parameter to the SettingsRegistry MergeSettings
This allows the MergeSettings function to write JSON data anchored
underneath the supplied anchor path.
Upgraded the SignalNotifiers calls in SetObject, MergeSettings and
MergeSettingsFileInternal to query the type of the merge value at the anchor path
and supply that as the type to the notification event.
Also the the above functions now supply the anchor key root as the
path that was modified instead of assuming root ""
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed whitespace inconsistencies in SettingsRegistryImpl
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added a queue for storing SignalNotifier calls when a thread is
currently signaling.
The queued calls are invoked by that thread after it has signaled it's
current queue of events
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* {lyn7251} Add material component example in Python
adds a AZ::Render::EditorMaterialComponent as an example of how to
override the default material from the scene building pipeline
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
* Only set the cube to a gray material
Skip loading the asset, instead just set the outPrefabAssetPath for the Prefab system to load
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
* stablizing the sub-id of procedural prefab groups
Signed-off-by: jackalbe <23512001+jackalbe@users.noreply.github.com>
The AZ_ENABLE_TRACING check was preventing the Archive System from using the sys_PakPriority value set from the CVar system
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
The EntityOutlinerListModel was violating the QAbstractItemModel contract in a few cases, as reported by `QAbstractItemModelTester`. The important ones causing issues were:
- Entry order was not guaranteed, leading to model indices pointing at invalid data
- Parent/child relationships could be temporarily invalid due to a change I made in EditorEntityModel::RemoveEntity to try to avoid an unnecessary reparent operation - as it turned out, the parent/child data was being cached even for recreated entities and not clearing child data could cause issues
- `EntityOutlinerListModel::ProcessEntityUpdates` was emitting data changed between two indices that didn't necessarily share a parent, which is [undefined behavior](https://doc.qt.io/qt-5/qabstractitemmodel.html#dataChanged)
The other reported issues (that weren't really causing issues with `QTreeView`) were:
- The root index had flags other than `Qt::ItemIsDropEnabled`
- `rowCount` showed all columns as having children
- `parent` showed indices as being parented to a non-0 column
This change introduces fixes for the above issues, namely:
- Reverts my change to `EditorEntityModel::RemoveEntity` to ensure we don't have invalid parent/child references sitting in the cache
- Ensures `EditorEntityModelEntry` child ordering is guaranteed sorted by EntityId, to prevent the `EntityOutlinerListModel` from having indices pointed at invalid data*.
- Fixes various model sanity issues, such as `rowCount` being 0 for indices with a non-0 column
Two unit tests were added to reproduce the invalid behavior and validate the fix: TestCreateFlatHierarchyUndoAndRedoWorks and TestCreateNestedHierarchyUndoAndRedoWorks
This change focuses on correctness over performance. My subjective in-Editor outliner experience is about the same, but it may be worthwhile to expand the test coverage with a benchmarking suite to look into areas for optimization.
*As a rough illustration of the previous child ordering behavior, consider the following entity hierarchy:
```
Root (EID 9999)
|_ Child1 (EID 2)
|_ Child2 (EID 3)
|_ Child3 (EID 4)
```
With an representations like the following pseudocode:
```
// EditorEntityModel representation
EditorEntityModelEntry root;
root.children[0] = 2;
root.children[1] = 3;
root.children[2] = 4;
// EditorOutlinerListModel representation
// row, column, user data (64 bit uint)
child1 = QModelIndex(0, 0, 2)
child2 = QModelIndex(1, 0, 3)
child3 = QModelIndex(2, 0, 4)
```
When removing a child, the `EditorEntityModel` used to do roughly the following:
```
// Swap and pop the last child
int indexToRemove = 0;
swap(root.children[indexToRemove], root.children[root.children.size() - 1]);
root.children.resize(root.children.size() - 1);
model.notifyRemoved(root, indexToRemove); // model removes the row indicated
// Leading to this EditorEntityModel state
root.children[0] = 4;
root.children[1] = 3;
// And this EntityOutlinerListModel state, note that the row indices are swapped from the indices in the backing storage
child2 = QModelIndex(0, 0, 3)
child3 = QModelIndex(1, 0, 4)
```
A QModelIndex having a row that doesn't match its underlying data is undefined behavior, and was the source of an intermittent crash in our `QSortFilterProxyModel` as subsequent updates to the wrong row led to an invalid proxy state.
Signed-off-by: nvsickle <nvsickle@amazon.com>
* fixes some warnings for newer versions of VS2022
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* more warning fixes
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* remove integ test filters from AzTest
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* remove integ test handling from AzTestRunner
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* changes integ tests of gridmate to regular tests and disables failing ones
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* removes the Integ from the EMotionFX tests, but leaves them disabled since they are failing
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* removes the Integ from the HttpRequestor tests and disables it since is not passing
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* changing integ tests for DISABLED, these ones are using files that are not there
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* fixes linux build
gridmate tests that were Integ are now disabled
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
* fixes linux warnings
Signed-off-by: Esteban Papp <81431996+amznestebanpapp@users.noreply.github.com>
santorac
Nicholas Van Sickle
Danilo Aimini
Steve Pham
chiyenteng
srikappa-amzn
SJ
Chris Burel
Vincent Liu
Alex Peterson
Allen Jackson
jonawals
amzn-mike
puvvadar
lumberyard-employee-dm
igarri
Pratik Patel
Yaakuro
hultonha
Esteban Papp