ATOM-15358 : Culling concurrency checker fails in AtomSampleViewer (merge from 1.0->main)

Inserting, updating, and removing entries from a VisibilityScene was made thread safe in a previous change, and now both the MeshFeatureProcessor and DiffuseProbeGridFeatureProcessor update entries at the same time from multiple threads. This leads to an assert in the Culling concurrency_checker, even though this is now valid behavior.

However, we still don't want to be adding, removing, or updating cullables between BeginCulling and EndCulling, which could cause a mismatch between the result of OctreeScene::GetEntryCount and the actual number of cullables in the scene.

-Added soft_lock_shared/soft_unlock_shared to the concurrency checker to allow multiple threads to acquire a lock when that is the desired behavior, while still asserting that nothing tries to acquire a shared lock when the concurrency checker is already locked.
-Update Culling.cpp to use the new soft_lock_shared when adding, updating, or removing cullables
-Added unit tests for the concurrency_checker
-Updated ArrayView unit test to use AZ_TEST_START_TRACE_SUPPRESSION instead of manually checking the assertion count, so that test now passes in release builds which do not assert.

Code/Framework/AtomCore/AtomCore/std/parallel/concurrency_checker.h

@@ -20,10 +20,12 @@
 
 namespace AZStd
 {
-    //! Simple class for verifying that no concurrent access is occuring.
+    //! Simple class for verifying that no concurrent access is occurring.
     //! This is *not* a synchronization primitive, and is intended simply for checking that no concurrency issues exist.
     //! It will be compiled out in release builds.
     //! Use concurrency_checker like a mutex (i.e. call soft_lock() and soft_unlock() around all instances of your data access).
+    //! Use soft_lock_shared and soft_unlock_shared around places where multiple threads are allowed to have read access
+    //! at the same time as long as nothing else already has a soft lock
     //! It will assert if there are multiple threads accessing the locked code/data at the same time.
     //! Expected use case is for defensive programming: when you do not expect any concurrent access within a system,
     //! but want to verify that it stays that way in the future, without incurring the overhead of a mutex.
@@ -34,7 +36,7 @@ namespace AZStd
         {
 #ifdef AZ_CONCURRENCY_CHECKER_ENABLED
             uint32_t count = ++m_concurrencyCounter;
-            AZ_Assert(count == 1, "Concurrency check failed. Multiple threads are trying to access data at the same time, or there is a lock/unlock mismatch.");
+            AZ_Assert(count == 1 && m_sharedConcurrencyCounter == 0, "Concurrency check failed. Multiple threads are trying to access data at the same time, or there is a lock/unlock mismatch.");
 #endif
         }
 
@@ -46,9 +48,27 @@ namespace AZStd
 #endif
         }
 
+        AZ_FORCE_INLINE void soft_lock_shared()
+        {
+#ifdef AZ_CONCURRENCY_CHECKER_ENABLED
+            AZ_Assert(m_concurrencyCounter == 0, "Concurrency check failed. A soft_lock_shared was attempted when there was already a soft_lock.");
+            ++m_sharedConcurrencyCounter;
+#endif
+        }
+
+        AZ_FORCE_INLINE void soft_unlock_shared()
+        {
+#ifdef AZ_CONCURRENCY_CHECKER_ENABLED
+            AZ_Assert(m_sharedConcurrencyCounter != 0, "Concurrency check failed. There is a shared_lock/shared_unlock mismatch.");
+            --m_sharedConcurrencyCounter;
+#endif
+        }
+
+
     private:
 #ifdef AZ_CONCURRENCY_CHECKER_ENABLED
         AZStd::atomic_uint32_t m_concurrencyCounter = 0;
+        AZStd::atomic_uint32_t m_sharedConcurrencyCounter = 0;
 #endif
     };