The problem was...
After a MaterialAsset reload, there could be two different versions of the MaterialAsset in memory: the old one and the reloaded one. The old one is still connected to buses and can send reinitialization messages when other things reload or reinitialize. So when the shader asset reloaded, both the old and new MaterialAsset were sending reinitialization messages. Material::OnMaterialAssetReinitialized was using the materialAsset parameter to initialize the Material, and the latest call to OnMaterialAssetReinitialized was for the *old* MaterialAsset.
The solution is to use the m_materialAsset member when reinitializing the Material. I also added checks in a couple places to skip unnecessary reinitialization, and added comments in the bus headers to warn developers about this issue.
Testing: Added a new step to ASV's MaterialHotReloadTest.bv.lua script for the error scenario, and this now passes. Ran ASV full test suite, both dx12 and vulkan, only known issues occurred.
ATOM-15728 Shader Hot Reload Fails in Debug Build
The main change was to add OnAssetReady handlers to each of the asset classes. See comments in ShaderAsset::OnAssetReady for a detailed explanation. In short, OnAssetReloaded gets missed while assets are being reloaded at the same time on multiple threads, but OnAssetReady is always called whenever connecting to the AssetBus because of its AssetConnectionPolicy.
The above change required the addition of a new AssetInitBus to call the PostLoadInit() functions. Because OnAssetReady connects to buses that are not mutex-protected, they have to be connected on the main thread. AssetInitBus::PostLoadInit is called every frame in RPISystem::SimulationTick. All Atom's asset handlers that need to do post-load initialization must connect to the AssetInitBus, and the asset will disconnect itself after initialization is complete.
We also need the Shader class to handle OnShaderAssetReinitialized to properly handle the shader reload.
With these changes I can click back and forth between "Blending On" and "Blending Off" many times (like 20 times) without issue.
This is specifically in support of:
ATOM-14613 Baseviewer MatertialHotReloadTest fails to change the color after turning blending on and off
ATOM-15728 Shader Hot Reload Fails in Debug Build
Chris Santora
alexpete