Made ShaderReloadDebugTracker store its static data in Environment system variables, so they are shared across dlls. This fixes issues with inconsistent indenting when debug operations are performed in different libraries.
New ShaderReloadDebugTracker operations in FullscreenTrianglePass.
Added a ShaderReloadDebugTracker message to Shader::GetVariant that includes asset built timestamp infromation, which I think will be really helpful in sorting out reload issues.
Renamed some functions and variables to remove a redundant "ShaderAsset" term.
Signed-off-by: santorac <55155825+santorac@users.noreply.github.com>
- Each resource type is tracked and updated separately
- Added caching ability for Raytracing srg to save ~2ms for a scene containing 100 x 50 vegetation patch
Signed-off-by: moudgils <moudgils@amazon.com>
* ATOM-16249 Adding draw srg caching to DynamicDrawContext since creating SRG can be expensive for some backend
Signed-off-by: Qing Tao <qingtao@amazon.com>
* Fix issues with seedlist for AutomatedTesting
Fixes error reporting so it will show the file hint in the tool.
Removes any missing assets from the .seed file.
Remove an unnecessary dependency from AutomatedTesting dependencies
file.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Make ArchiveComponent use AZ::IO::IArchive
Initial changes that will get the sychronous calls in ArchiveComponent
to use IArchive interface rather than external zip/7z tools.
Some of the asynchronous api are still in place, anything that wasn't
being used has been removed for now. This may change later if we move
towards all the api being asynchronous. Until then, we can't remove the
reliance upon the external archive tools completely.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Updates AZStd::thread constructors
Adds a variadic constructor which forwards args to the functor.
Because of our thread_desc extension, there was confusion on the
arugments, so the args were reordered to take the thread_desc first,
before the functor and args.
Also the thread_desc is taken as reference rather than by pointer.
Update callsites to account for this change.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Async operation of ArchiveComponent api
This sets up the ArchiveComponent to operate asynchronously. It uses
promise/future to transfer results to caller.
This is still broken, there's a few things that need to get fixed up,
but this is a good checkpoint for the work as it solidifies the api,
cleans up a bunch of unused code, and compiles.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes the platform-specific ArchiveComponen
These are no longer needed, as they control the direct interaction with
host OS tools like 7za.exe or /bin/zip.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes the platform-specific files from cmake
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes the 7za.exe (and legal notice)
This tool is no longer needed in the repo.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fixes usage of IArchive::GetFullPath()
This changed to return a PathView, updated to reflect that.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix promises and threads
Make sure promises are only set exactly once. This meant reworking some
of the initial error checking.
Detach threads when created. Adds [[nodiscard]] to the functions that
return a future. Since threads are detached, the future is the main way
to get communication from the thread.
Clean up interface, add comments.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* More edits to thread creation
Changes to thread construction to account for parameter change.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix some remaining issues with ArchiveComponent
Put created threads inside a container, then join them at Deactivate.
Fix asset bundler case when injecting a file with no working directory.
Fix thread constructor that applies args to a function.
Fix lambdas to take string args by value rather than reference.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fixes some remaining bugs in ArchiveComponent
Open archive as read-only during extract & list operations.
Fixes paths issues.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix initialize of opaque thread handle in thread_UnixLike.h
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed unused variable in AssetBundleComponent.cpp to fix compiler warning
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix some issues with archives
File paths in the CDR and the local headers need to match, but there
were issues with path separators and case that made it possible to get
invalid headers errors in some archives.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Adds some new ArchiveComponent unit tests
Adds new tests for extraction of archive and adding files from a file
list to an archive.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix file data offset issues when opening archives
When opening an INestedArchive it would run through the CDR headers to
create file entries in the zip cache. The offsets to the compressed
data were being calculated incorrectly because they were using the CDR
headers rather than jumping to the local file headers and getting
offsets from those sizes.
Removed and refactored some archive validation flow and zip cache
factory init methods to either init default or init w/ additional
validation checks.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Addresses PR feedback
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address more points of feedback in PR
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address additional PR feedback
Fixes up some error checks and uses of strings vs paths.
Enable archive component tests on Linux so we can see if they will
pass.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address PR feedback
Change the INestedArchive interface to list files as AZ::IO::Path.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Disabling the ArchiveComponent tests on Linux
They failed so we will revisit them to attempt a fix.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Rename a member variable to be more accurate
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address feedback on PR
Bump version of Archive Components for serialize context.
Improve error messages during archive open and validation.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Revert recent changes
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
- Added new interface type AZ::Debug::Profiler to externally register profiler systems
- Modified the Atom CPU profiler to register as an AzCore profiler
-- This allows full engine markers to be visualized in the associated ImGui tool
- Converted all AZ_ATOM_PROFILE_* macros to use AZ_PROFILE_* macros instead
Signed-off-by: AMZN-ScottR 24445312+AMZN-ScottR@users.noreply.github.com
* Fix loading of PipelineLibraries from disk for DX12 backend.
Signed-off-by: moudgils <moudgils@amazon.com>
* Disabled Saving out PipelineLibraries (for DX12) if pix or Renderdoc is enabled.
Addressed some feedback
Signed-off-by: moudgils <moudgils@amazon.com>
* Fixed an issue withe loading PipelineLibraries and added a cleaner abstraction to not save empty libraries for dx12
Signed-off-by: moudgils <moudgils@amazon.com>
Merge pull request #3574 from aws-lumberyard-dev/Atom/santorac/MoreShaderHotReloadFixes
Fixed a shader hot reload issue where the new root shader variant asset was not getting saved in the Shader object during OnAssetReloaded, it was only saved during OnAssetReady.
Fixed shader variant hot reload which was failing due to mismatched timestamps. The ShaderAsset was using microseconds and the ShaderVariantAsset was using system ticks. Since ticks will always be higher than microseconds, stale variants were not prevented from being used.
AtomSampleViewer's MaterialHotReloadTest script is passing again.
NOTE: The memory driller is still intact for now to avoid needing to
modify allocators, but the frame/cpu portions of driller and the
standalone executable are now gone.
Signed-off-by: Jeremy Ong <jcong@amazon.com>
* Metal sync interval support
Add shader name to SRGPool for debugging purposes
Set default value of rpi_vsync_interval to 1
Signed-off-by: moudgils <moudgils@amazon.com>
* Final update copyright headers to reference license files at the repo root
Signed-off-by: spham <spham@amazon.com>
* Fix copyright validator unit tests to support the stale O3DE header scenario
Signed-off-by: spham <spham@amazon.com>
ATOM-15837 Remove Unused Shader System Related *2 Classes From Stabilization Branch.
Testing: ASV full test suite on dx12 and vulkan. Only saw known issues.
Added a ShaderSystem supervariant to provide a system-wide supervariant name.
Changed ShaderAsset to append the system-wide supervariant name when searching for supervariants.
Added the NoMSAA supervariant to several shaders.
* Fixed organization of the AssetProcessor SourceAssetBrowser
Assets within the Engine Root were grouped under a '/' entry.
That has been fixed to use the relative path within the engine root for
those assets
Assets outside of the Engine Root, but on the same drive were using
absolute paths before. Now there are child entries that navigate up the
directory hierarchy to those asset locations
* Added ly_enable_gems call to Atom gems targets that are required
The DefaultLevel.prefab contains several Atom components, that require
the Atom RHI, RPI, Common_Feature, ShaderBuilder and AtomLyIntegration CommonFeatures
gems to be enabled in order to successfully process in the
AssetProcessor.
* Added ly_enable_gems call to make the Camera gem required in Tools,
Builders and Clients.
This is needed as the DefaultLevel.prefab contains an Editor Camera
Component
* Adding the ly_enable_gem call to make the Maestro gem required
CrySystem currently requires Maestro to be enabled in order to
initialize
* Added ly_enable_gems call to the SceneProcessing gem to make it required
The SceneCore and SceneData libraries that are part of the core engine
Code folder requires the SceneProcessing gem to be enabled in order to
invoke the InitializeDynamicModule hooks in DllMain.cpp in order to
initialize those libraries.
* Fixed bad argument in comment for Prefab CMakeLists.txt
* Fixed Assert in Asset Builders due to the Atom RPI Builder
The Atom RPI Builder was enabling the Asset Catalog for the ScriptAsset a second time
The Atom Feature Common EditorSystemCommonComponent.cpp which also loads
in the AssetBuilder is enabling the Asset Catalog for the ScriptAsset
Added BehaviorContext reflection to the OutputDeviceTransformType enum
to fix the BehaviorContext errors about reflecting a method that returns
such an enum
* Added TypeId output to the JsonDeserializer report message about missing
ClassData
Previously the report callback would indicate that the target type was
missing Serialization class data, but didn't indicate the TypeId of the
target type
* Added support to the ly_enable_gems function to be able to support
0 gems being enabled.
Updated the Install step for CMake to propagate any ly_enable_gems
within a CMakeLists.txt for a target into the generated CMakeLists.txt
that is made for each installed IMPORTED target
* Adding newline to the end of the Camera Gem CMakeLists.txt
* Fixing target TYPE parameter for actual Gem Modules to use the GEM_MODULE tag instead of MODULE
* Reverting change to the DESTINATION directory for the installed CMakeLists.txt to use the relative path to the installed directory
* Adding the Atom_Bootstrap gem as a required gem
The Client and GameLaunchers required the Atom_Bootstrap gem in order to create the NativeWindow
Added Atom_Feature_Common client module as a runtime dependency of the AtomLyIntegration CommonsFeature client module
* Fixed register.py --all-projects-path and --all-gems-path arguments to
NOT register projects or gems that are within a template folder
Fixed reading of old pre-1.0 o3de_manifest.json files where the
"engines" key was a json array
* Changed how the relative target source directory is calculated when that source directroy resides outside of the engine root.
The final dirname component is used with a unique SHA256 has to form a <dirname>-<8 char SHA256> folder for installing files into
* Adding newline to the end of Atom_Bootstrap CMakeLists.txt
* Moving ly_enable_gems variants for Tools and Builders inside of PAL_TRAIT_BUILD_HOST_TOOLS block
* Adding a comment to AWSCore.ResourceMappingTool target to indicate that it is not a GEM_MODULE.
Furthermore it cannot be loaded with the Gem system because the library is in a different directory the executable
[ATOM-14544] Add Reset() function to IShaderVariantFinder
ShaderVariantAsyncLoader::Reset() now Shutdown and Init(). It clears
the cache of ShaderVariantAssets it keeps in memory.
Signed-off-by: garrieta <garrieta@amazon.com>
The problem was...
After a MaterialAsset reload, there could be two different versions of the MaterialAsset in memory: the old one and the reloaded one. The old one is still connected to buses and can send reinitialization messages when other things reload or reinitialize. So when the shader asset reloaded, both the old and new MaterialAsset were sending reinitialization messages. Material::OnMaterialAssetReinitialized was using the materialAsset parameter to initialize the Material, and the latest call to OnMaterialAssetReinitialized was for the *old* MaterialAsset.
The solution is to use the m_materialAsset member when reinitializing the Material. I also added checks in a couple places to skip unnecessary reinitialization, and added comments in the bus headers to warn developers about this issue.
Testing: Added a new step to ASV's MaterialHotReloadTest.bv.lua script for the error scenario, and this now passes. Ran ASV full test suite, both dx12 and vulkan, only known issues occurred.
This was done while working on "ATOM-15728 Shader Hot Reload Fails in Debug Build", but it turned out these changes did not actually fix the issue (or any other known hot-reload issue). Still, these improvements are appropriate as they correct logical oversights.
ShaderVariant was not listening to asset reloads. It needs to know when the ShaderVariantAsset reload happens so it can reinitialize it's members as well as propagate reinitialization messages. I added a member for the ShaderAsset as the class needs this to reinitialize itself. So now the class listens for reloads of both the ShaderVariantAsset and the ShaderAsset.
Shader was not listening for ShaderAsset reinitialization events.
Updated the API for ShaderReloadNotificationBus's OnShaderVariantReinitialized to include the ShaderVariant which is the most relevant information (the other information wasn't really being used anyway).
ATOM-15728 Shader Hot Reload Fails in Debug Build
The main change was to add OnAssetReady handlers to each of the asset classes. See comments in ShaderAsset::OnAssetReady for a detailed explanation. In short, OnAssetReloaded gets missed while assets are being reloaded at the same time on multiple threads, but OnAssetReady is always called whenever connecting to the AssetBus because of its AssetConnectionPolicy.
The above change required the addition of a new AssetInitBus to call the PostLoadInit() functions. Because OnAssetReady connects to buses that are not mutex-protected, they have to be connected on the main thread. AssetInitBus::PostLoadInit is called every frame in RPISystem::SimulationTick. All Atom's asset handlers that need to do post-load initialization must connect to the AssetInitBus, and the asset will disconnect itself after initialization is complete.
We also need the Shader class to handle OnShaderAssetReinitialized to properly handle the shader reload.
With these changes I can click back and forth between "Blending On" and "Blending Off" many times (like 20 times) without issue.
This is specifically in support of:
ATOM-14613 Baseviewer MatertialHotReloadTest fails to change the color after turning blending on and off
ATOM-15728 Shader Hot Reload Fails in Debug Build
* [ATOM-15472] Shader Build Pipeline: Remove Deprecated Files And Functions That
Predate The Shader Supervariants
These are the essential impactful changes as a result of deprecating the
ShaderResourceGroupAsset.
* Addressed feedback by @moudgils. Better comments in header files.
* More updates related with deprecation of ShaderResourceGroupAsset
* Deleted the temporary version 2 classes.
* Updated version of the shader asset builders.
* Updated version of all the shader related classes impacted
by the Supervariant concept and deprecation of ShaderResourceGroupAsset
* Changes to *.pass and DGI, Reflections and RayTracing.
* changes to material related assets
* changes to core lights
* Changes to auxgeom/dynamic draw.
* changes to decals, lyshine, imguipass
* changes to RPI Pass classes
* Shader for SceneSrg, ViewSrg and ForwardPass Srgs.
* changes to mesh, skinned mesh, Morphtarget.
* Fixes to RayTracingPass.cpp & now allow empty srg in shaders.
* Updated Atom_RPI.Tests
* Simplified InstanceDatabase by removing AddHandler
------------------------------------------------------------------------------------
* Updated DiffuseGI precompiled shaders.
Added RayTracingSceneSrg and RayTracingMaterialSrg shader asset.
Updated ShaderAssetCreator::Clone to handle the supervariant when processing root variants.
Co-authored-by: Doug McDiarmid <dmcdiar@amazon.com>
------------------------------------------------------------------------------------
* Changed semantics for some PassSrg to SRG_PerPass_WithFallback.
AuxGeom/FixedShapeProcessor.cpp requires SRG_PerDraw on ObjectSrg.
Removed names of SceneSrg and ViewSrg from RPISystemDescriptor.cpp
* Moved ShaderLib/Atom/Features/DummyEntryFunctions.azsli
To Gems/Atom/RPI/Assets/ShaderLib/Atom/RPI/DummyEntryFunctions.azsli
Removed redundant checking for finalization in
ShaderResourceGroupLayout.cpp
* Fixed race condition bug for Shader::FindOrCreate.
InstanceDatabase<>::CreateInstance() needs to be atomic
for instance creation and initialization.
Added optional InstanceHandler::CreateFunctionWithParams to accomodate
to the needs of Instances that need more than an asset reference
to be able to be created an initialzed.
Removed ShaderResourceGroup::FindOrCreate() only ::Create is available
now.
* Renamed scene_and_view_srgs.* as SceneAndViewSrgs.*
Changed GetAzslFileOfOrigin for GetUniqueId
* Fixed unit tests.
* Reverted the serialization name of m_uniqueId back to
"m_azslFileOfOrigin" so precompiled shaders don't fail
in layout comparison.
* Fixed AtomCore.Tests
Removed non-applicable test. InstanceDatabase.AddHandler() is not
available anymore.
* The Null rhi is re-enabled for shader compilation.
Signed-off-by: garrieta <garrieta@amazon.com>
This is a temporary fix, in the future ShaderVariantAsyncLoader will use
OnCatalogAssetRemoved()/ OnCatalogAssetAdded().
Signed-off-by: garrieta <garrieta@amazon.com>
* [ATOM-15276] Shader Build Pipeline: Add Shader Supervariant System.
Added ShaderAssetBuilder2 & ShaderVariantAssetBuilder2.
Added ShaderAsset2, ShaderVariantAsset2.
Eventually they will be the only builders. AzslBuilder &
SrgLayoutBuilder will be removed.
ShaderResourceGroupAsset will be removed. ShaderAssetBuilder &
ShaderVariantAssetBuilder will be replaced.
Signed-off-by: garrieta <garrieta@amazon.com>
santorac
moudgils
Qing Tao
amzn-phist
bosnichd
Scott Romero
Mike Balfour
Esteban Papp
Jeremy Ong
chcurran
Steve Pham
Chris Galvan
Gene Walters
Chris Santora
Doug McDiarmid
nvsickle
lumberyard-employee-dm
galibzon
mnaumov
AMZN-mnaumov
alexpete