Archive Component - Rewrite and additional work on Archive and Asset Bundler (#4332)

* Fix issues with seedlist for AutomatedTesting

Fixes error reporting so it will show the file hint in the tool.
Removes any missing assets from the .seed file.
Remove an unnecessary dependency from AutomatedTesting dependencies
file.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Make ArchiveComponent use AZ::IO::IArchive

Initial changes that will get the sychronous calls in ArchiveComponent
to use IArchive interface rather than external zip/7z tools.

Some of the asynchronous api are still in place, anything that wasn't
being used has been removed for now.  This may change later if we move
towards all the api being asynchronous.  Until then, we can't remove the
reliance upon the external archive tools completely.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Updates AZStd::thread constructors

Adds a variadic constructor which forwards args to the functor.
Because of our thread_desc extension, there was confusion on the
arugments, so the args were reordered to take the thread_desc first,
before the functor and args.
Also the thread_desc is taken as reference rather than by pointer.
Update callsites to account for this change.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Async operation of ArchiveComponent api

This sets up the ArchiveComponent to operate asynchronously.  It uses
promise/future to transfer results to caller.

This is still broken, there's a few things that need to get fixed up,
but this is a good checkpoint for the work as it solidifies the api,
cleans up a bunch of unused code, and compiles.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Removes the platform-specific ArchiveComponen

These are no longer needed, as they control the direct interaction with
host OS tools like 7za.exe or /bin/zip.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Removes the platform-specific files from cmake

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Removes the 7za.exe (and legal notice)

This tool is no longer needed in the repo.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fixes usage of IArchive::GetFullPath()

This changed to return a PathView, updated to reflect that.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix promises and threads

Make sure promises are only set exactly once.  This meant reworking some
of the initial error checking.
Detach threads when created.  Adds [[nodiscard]] to the functions that
return a future.  Since threads are detached, the future is the main way
to get communication from the thread.
Clean up interface, add comments.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* More edits to thread creation

Changes to thread construction to account for parameter change.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix some remaining issues with ArchiveComponent

Put created threads inside a container, then join them at Deactivate.
Fix asset bundler case when injecting a file with no working directory.
Fix thread constructor that applies args to a function.
Fix lambdas to take string args by value rather than reference.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fixes some remaining bugs in ArchiveComponent

Open archive as read-only during extract & list operations.
Fixes paths issues.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix initialize of opaque thread handle in thread_UnixLike.h

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>

* Removed unused variable in AssetBundleComponent.cpp to fix compiler warning

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>

* Fix some issues with archives

File paths in the CDR and the local headers need to match, but there
were issues with path separators and case that made it possible to get
invalid headers errors in some archives.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Adds some new ArchiveComponent unit tests

Adds new tests for extraction of archive and adding files from a file
list to an archive.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix file data offset issues when opening archives

When opening an INestedArchive it would run through the CDR headers to
create file entries in the zip cache.  The offsets to the compressed
data were being calculated incorrectly because they were using the CDR
headers rather than jumping to the local file headers and getting
offsets from those sizes.

Removed and refactored some archive validation flow and zip cache
factory init methods to either init default or init w/ additional
validation checks.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Addresses PR feedback

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address more points of feedback in PR

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address additional PR feedback

 Fixes up some error checks and uses of strings vs paths.
 Enable archive component tests on Linux so we can see if they will
 pass.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address PR feedback

Change the INestedArchive interface to list files as AZ::IO::Path.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Disabling the ArchiveComponent tests on Linux

They failed so we will revisit them to attempt a fix.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Rename a member variable to be more accurate

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address feedback on PR

Bump version of Archive Components for serialize context.
Improve error messages during archive open and validation.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Revert recent changes

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>

Code/Framework/AzFramework/AzFramework/Application/Application.cpp

@@ -479,14 +479,16 @@ namespace AzFramework
         newThreadDesc.m_cpuId = AFFINITY_MASK_USERTHREADS;
         newThreadDesc.m_name = newThreadName;
         AZStd::binary_semaphore binarySemaphore;
-        AZStd::thread newThread([&workForNewThread, &binarySemaphore, &newThreadName]
-        {
-            AZ_PROFILE_SCOPE(AzFramework,
-                "Application::PumpSystemEventLoopWhileDoingWorkInNewThread:ThreadWorker %s", newThreadName);
+        AZStd::thread newThread(
+            newThreadDesc,
+            [&workForNewThread, &binarySemaphore, &newThreadName]
+            {
+                AZ_PROFILE_SCOPE(AzFramework,
+                    "Application::PumpSystemEventLoopWhileDoingWorkInNewThread:ThreadWorker %s", newThreadName);
 
-            workForNewThread();
-            binarySemaphore.release();
-        }, &newThreadDesc);
+                workForNewThread();
+                binarySemaphore.release();
+            });
         while (!binarySemaphore.try_acquire_for(eventPumpFrequency))
         {
             PumpSystemEventLoopUntilEmpty();

Code/Framework/AzFramework/AzFramework/Archive/Archive.cpp

@@ -1631,7 +1631,20 @@ namespace AZ::IO
             return nullptr;
         }
 
-        ZipDir::CacheFactory factory(ZipDir::ZD_INIT_FAST, nFactoryFlags);
+        ZipDir::InitMethod initType = ZipDir::InitMethod::Default;
+        if (!ZipDir::IsReleaseConfig)
+        {
+            if ((nFlags & INestedArchive::FLAGS_FULL_VALIDATE) != 0)
+            {
+                initType = ZipDir::InitMethod::FullValidation;
+            }
+            else if ((nFlags & INestedArchive::FLAGS_VALIDATE_HEADERS) != 0)
+            {
+                initType = ZipDir::InitMethod::ValidateHeaders;
+            }
+        }
+
+        ZipDir::CacheFactory factory(initType, nFactoryFlags);
 
         ZipDir::CachePtr cache = factory.New(szFullPath->c_str());
         if (cache)

Code/Framework/AzFramework/AzFramework/Archive/INestedArchive.h

@@ -11,7 +11,9 @@
 
 #include <AzCore/IO/Path/Path_fwd.h>
 #include <AzCore/Math/Crc.h>
+#include <AzCore/std/containers/vector.h>
 #include <AzCore/std/smart_ptr/intrusive_base.h>
+#include <AzCore/std/string/string.h>
 #include <AzFramework/Archive/Codec.h>
 
 namespace AZ::IO
@@ -71,6 +73,13 @@ namespace AZ::IO
             // multiple times
             FLAGS_DONT_COMPACT = 1 << 5,
 
+            // if this is set, validate header data when opening the archive
+            FLAGS_VALIDATE_HEADERS = 1 << 9,
+
+            // if this is set, validate header data when opening the archive and validate CRCs when decompressing
+            // & reading files.
+            FLAGS_FULL_VALIDATE = 1 << 10,
+
             // Disable a pak file without unloading it, this flag is used in combination with patches and multiplayer
             // to ensure that specific paks stay in the position(to keep the same priority) but being disabled
             // when running multiplayer
@@ -128,6 +137,10 @@ namespace AZ::IO
         //   Deletes all files and directories in the archive.
         virtual int RemoveAll() = 0;
 
+        // Summary:
+        //   Lists all the files in the archive.
+        virtual int ListAllFiles(AZStd::vector<AZ::IO::Path>& outFileEntries) = 0;
+
         // Summary:
         //   Finds the file; you don't have to close the returned handle.
         // Returns:

Code/Framework/AzFramework/AzFramework/Archive/NestedArchive.cpp

@@ -89,11 +89,51 @@ namespace AZ::IO
         return m_pCache->RemoveDir(fullPath);
     }
 
+    //////////////////////////////////////////////////////////////////////////
     int NestedArchive::RemoveAll()
     {
         return m_pCache->RemoveAll();
     }
 
+    //////////////////////////////////////////////////////////////////////////
+    // Helper for 'ListAllFiles' to recursively traverse the FileEntryTree and gather all the files
+    void EnumerateFilesRecursive(AZ::IO::Path currentPath, ZipDir::FileEntryTree* currentTree, AZStd::vector<AZ::IO::Path>& fileList)
+    {
+        // Drill down directories first...
+        for (auto dirIter = currentTree->GetDirBegin(); dirIter != currentTree->GetDirEnd(); ++dirIter)
+        {
+            if (ZipDir::FileEntryTree* subTree = currentTree->GetDirEntry(dirIter);
+                subTree != nullptr)
+            {
+                EnumerateFilesRecursive(currentPath / currentTree->GetDirName(dirIter), subTree, fileList);
+            }
+        }
+
+        // Then enumerate the files in current directory...
+        for (auto fileIter = currentTree->GetFileBegin(); fileIter != currentTree->GetFileEnd(); ++fileIter)
+        {
+            fileList.emplace_back(currentPath / currentTree->GetFileName(fileIter));
+        }
+    }
+
+    //////////////////////////////////////////////////////////////////////////
+    // lists all files in the archive
+    int NestedArchive::ListAllFiles(AZStd::vector<AZ::IO::Path>& outFileEntries)
+    {
+        AZStd::vector<AZ::IO::Path> filesInArchive;
+
+        ZipDir::FileEntryTree* tree = m_pCache->GetRoot();
+        if (!tree)
+        {
+            return ZipDir::ZD_ERROR_UNEXPECTED;
+        }
+
+        EnumerateFilesRecursive(AZ::IO::Path{ AZ::IO::PosixPathSeparator }, tree, filesInArchive);
+
+        AZStd::swap(outFileEntries, filesInArchive);
+        return ZipDir::ZD_ERROR_SUCCESS;
+    }
+
     //////////////////////////////////////////////////////////////////////////
     // Adds a new file to the zip or update an existing one
     // adds a directory (creates several nested directories if needed)

Code/Framework/AzFramework/AzFramework/Archive/NestedArchive.h

@@ -39,7 +39,7 @@ namespace AZ::IO
 
         NestedArchive(IArchive* pArchive, AZStd::string_view strBindRoot, ZipDir::CachePtr pCache, uint32_t nFlags = 0);
         ~NestedArchive() override;
-        
+
         auto GetRootFolderHandle() -> Handle override;
 
         // Adds a new file to the zip or update an existing one
@@ -68,6 +68,9 @@ namespace AZ::IO
         // deletes all files from the archive
         int RemoveAll() override;
 
+        // lists all the files in the archive
+        int ListAllFiles(AZStd::vector<AZ::IO::Path>& outFileEntries) override;
+
         // finds the file; you don't have to close the returned handle
         Handle FindFile(AZStd::string_view szRelativePath) override;
 
@@ -79,7 +82,6 @@ namespace AZ::IO
 
         // returns the full path to the archive file
         AZ::IO::PathView GetFullPath() const override;
-        ZipDir::Cache* GetCache();
 
         uint32_t GetFlags() const override;
         bool SetFlags(uint32_t nFlagsToSet) override;
@@ -87,12 +89,15 @@ namespace AZ::IO
 
         bool SetPackAccessible(bool bAccessible) override;
 
+        ZipDir::Cache* GetCache();
+
     protected:
         // returns the pointer to the relative file path to be passed
         // to the underlying Cache pointer. Uses the given buffer to construct the path.
         // returns nullptr if the file path is invalid
         AZ::IO::FixedMaxPathString AdjustPath(AZStd::string_view szRelativePath);
 
+
         ZipDir::CachePtr m_pCache;
         // the binding root may be empty string - in this case, the absolute path binding won't work
         AZ::IO::Path m_strBindRoot;

Code/Framework/AzFramework/AzFramework/Archive/ZipDirCache.cpp

@@ -101,10 +101,11 @@ namespace AZ::IO::ZipDir
         FileEntry* operator -> () { return m_pFileEntry; }
         FileEntryTransactionAdd(Cache* pCache, AZStd::string_view szRelativePath)
             : m_pCache(pCache)
+            , m_szRelativePath(AZ::IO::PosixPathSeparator)
             , m_bCommitted(false)
         {
             // Update the cache string pool with the relative path to the file
-            auto pathIt = m_pCache->m_relativePathPool.emplace(AZ::IO::PathView(szRelativePath).LexicallyNormal());
+            auto pathIt = m_pCache->m_relativePathPool.emplace(AZ::IO::PathView(szRelativePath, AZ::IO::PosixPathSeparator).LexicallyNormal());
             m_szRelativePath = *pathIt.first;
             // this is the name of the directory - create it or find it
             m_pFileEntry = m_pCache->GetRoot()->Add(m_szRelativePath.Native());
@@ -740,6 +741,16 @@ namespace AZ::IO::ZipDir
                 {
                     return ZD_ERROR_CORRUPTED_DATA;
                 }
+                if (pFileEntry->bCheckCRCNextRead)
+                {
+                    pFileEntry->bCheckCRCNextRead = false;
+                    uLong uCRC32 = AZ::Crc32((Bytef*)pUncompressed, nSizeUncompressed);
+                    if (uCRC32 != pFileEntry->desc.lCRC32)
+                    {
+                        AZ_Warning("Archive", false, "ZD_ERROR_CRC32_CHECK: Uncompressed stream CRC32 check failed");
+                        return ZD_ERROR_CRC32_CHECK;
+                    }
+                }
             }
         }
 

Code/Framework/AzFramework/AzFramework/Archive/ZipDirCacheFactory.cpp

@@ -29,7 +29,7 @@ namespace AZ::IO::ZipDir
     // this sets the window size of the blocks of data read from the end of the file to find the Central Directory Record
     // since normally there are no
     static constexpr size_t CDRSearchWindowSize = 0x100;
-    CacheFactory::CacheFactory(InitMethodEnum nInitMethod, uint32_t nFlags)
+    CacheFactory::CacheFactory(InitMethod nInitMethod, uint32_t nFlags)
     {
         m_nCDREndPos = 0;
         m_bBuildFileEntryMap = false; // we only need it for validation/debugging
@@ -448,7 +448,6 @@ namespace AZ::IO::ZipDir
     // builds up the m_mapFileEntries
     bool CacheFactory::BuildFileEntryMap()
     {
-
         Seek(m_CDREnd.lCDROffset);
 
         if (m_CDREnd.lCDRSize == 0)
@@ -530,14 +529,6 @@ namespace AZ::IO::ZipDir
             {
                 // Add this file entry.
                 char* str = reinterpret_cast<char*>(pFileName);
-                for (int i = 0; i < pFile->nFileNameLength; i++)
-                {
-                    str[i] = std::tolower(str[i], std::locale());
-                    if (str[i] == AZ_WRONG_FILESYSTEM_SEPARATOR)
-                    {
-                        str[i] = AZ_CORRECT_FILESYSTEM_SEPARATOR;
-                    }
-                }
                 str[pFile->nFileNameLength] = 0; // Not standard!, may overwrite signature of the next memory record data in zip.
                 AddFileEntry(str, pFile, extra);
             }
@@ -574,11 +565,7 @@ namespace AZ::IO::ZipDir
 
         FileEntryBase fileEntry(*pFileHeader, extra);
 
-        // when using encrypted headers we should always initialize data offsets from CDR
-        if ((m_encryptedHeaders != ZipFile::HEADERS_NOT_ENCRYPTED || m_nInitMethod >= ZD_INIT_FULL) && pFileHeader->desc.lSizeCompressed)
-        {
-            InitDataOffset(fileEntry, pFileHeader);
-        }
+        InitDataOffset(fileEntry, pFileHeader);
 
         if (m_bBuildFileEntryMap)
         {
@@ -606,142 +593,81 @@ namespace AZ::IO::ZipDir
         {
             Seek(pFileHeader->lLocalHeaderOffset);
 
-            // read the local file header and the name (for validation) into the buffer
-            AZStd::vector<char>pBuffer;
-            uint32_t nBufferLength = sizeof(ZipFile::LocalFileHeader) + pFileHeader->nFileNameLength;
-            pBuffer.resize(nBufferLength);
-            Read(&pBuffer[0], nBufferLength);
+            // Read only the LocalFileHeader w/ no additional bytes ('name' or 'extra' fields)
+            AZStd::vector<char> buffer;
+            uint32_t bufferLen = sizeof(ZipFile::LocalFileHeader);
+            buffer.resize_no_construct(bufferLen);
+            Read(buffer.data(), bufferLen);
 
-            // validate the local file header (compare with the CDR file header - they should contain basically the same information)
-            const auto* pLocalFileHeader = reinterpret_cast<const ZipFile::LocalFileHeader*>(&pBuffer[0]);
-            if (pFileHeader->desc != pLocalFileHeader->desc
-                || pFileHeader->nMethod != pLocalFileHeader->nMethod
-                || pFileHeader->nFileNameLength != pLocalFileHeader->nFileNameLength
-                // for a tough validation, we can compare the timestamps of the local and central directory entries
-                // but we won't do that for backward compatibility with ZipDir
-                //|| pFileHeader->nLastModDate != pLocalFileHeader->nLastModDate
-                //|| pFileHeader->nLastModTime != pLocalFileHeader->nLastModTime
-                )
+            const auto* localFileHeader = reinterpret_cast<const ZipFile::LocalFileHeader*>(buffer.data());
+
+            // set the correct file data offset...
+            fileEntry.nFileDataOffset = pFileHeader->lLocalHeaderOffset + sizeof(ZipFile::LocalFileHeader) +
+                localFileHeader->nFileNameLength + localFileHeader->nExtraFieldLength;
+
+            fileEntry.nEOFOffset = fileEntry.nFileDataOffset + fileEntry.desc.lSizeCompressed;
+
+            if (m_nInitMethod != ZipDir::InitMethod::Default)
             {
-                AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED:"
-                    " The local file header descriptor doesn't match the basic parameters declared in the global file header in the file."
-                    " The archive content is misconsistent and may be damaged. Please try to repair the archive");
-                return;
+                if (m_nInitMethod == ZipDir::InitMethod::FullValidation)
+                {
+                    // Mark the FileEntry to check CRC when the next read occurs
+                    fileEntry.bCheckCRCNextRead = true;
+                }
+
+                // Timestamps
+                if (pFileHeader->nLastModDate != localFileHeader->nLastModDate
+                    || pFileHeader->nLastModTime != localFileHeader->nLastModTime)
+                {
+                    AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED: (%s)\n"
+                        " The local file header's modification timestamps don't match that of the global file header in the archive."
+                        " The archive timestamps are inconsistent and may be damaged. Check the archive file.", m_szFilename.c_str());
+                    // don't return here, it may be ok.
+                }
+
+                // Validate data
+                if (pFileHeader->desc != localFileHeader->desc  // this checks CRCs and compressed/uncompressed sizes
+                    || pFileHeader->nMethod != localFileHeader->nMethod
+                    || pFileHeader->nFileNameLength != localFileHeader->nFileNameLength)
+                {
+                    AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED: (%s)\n"
+                        " The local file header descriptor doesn't match basic parameters declared in the global file header in the file."
+                        " The archive content is inconsistent and may be damaged. Please try to repair the archive.", m_szFilename.c_str());
+                    // return here because further checks aren't worse than this.
+                    return;
+                }
+
+                // Read extra data
+                uint32_t extraDataLen = localFileHeader->nFileNameLength + localFileHeader->nExtraFieldLength;
+                buffer.resize_no_construct(buffer.size() + extraDataLen);
+                Read(buffer.data() + buffer.size(), extraDataLen);
+
+                // Compare local file name with the CDR file name, they should match
+                AZStd::string_view zipFileName{ buffer.data() + sizeof(ZipFile::LocalFileHeader), localFileHeader->nFileNameLength };
+                AZStd::string_view cdrFileName{ reinterpret_cast<const char*>(pFileHeader + 1), pFileHeader->nFileNameLength };
+                if (zipFileName != cdrFileName)
+                {
+                    AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED: (%s)\n"
+                        " The file name in the local file header doesn't match the name in the global file header."
+                        " The archive content is inconsisten with the directory. Please check the archive.", m_szFilename.c_str());
+                }
+
+                // CDR and local "extra field" lengths may be different, should we compare them if they are equal?
+
+                // make sure it's the same file and the fileEntry structure is properly initialized
+                AZ_Assert(fileEntry.nFileHeaderOffset == pFileHeader->lLocalHeaderOffset,
+                    "The file entry header offset doesn't match the file header local offst (%s)", m_szFilename.c_str());
+
+                if (fileEntry.nFileDataOffset >= m_nCDREndPos)
+                {
+                    AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED: (%s)\n"
+                        " The global file header declares the file which crosses the boundaries of the archive."
+                        " The archive is either corrupted or truncated, please try to repair it", m_szFilename.c_str());
+                }
+
+                // End Validation
             }
-
-            // now compare the local file name with the one recorded in CDR: they must match.
-            auto CompareNoCase = [](const char lhs, const char rhs) { return std::tolower(lhs, std::locale()) == std::tolower(rhs, std::locale()); };
-            auto zipFileDataBegin = pBuffer.begin() + sizeof(ZipFile::LocalFileHeader);
-            auto zipFileDataEnd = zipFileDataBegin + pFileHeader->nFileNameLength;
-            if (!AZStd::equal(zipFileDataBegin, zipFileDataEnd, reinterpret_cast<const char*>(pFileHeader + 1), CompareNoCase))
-            {
-                // either file name, or the extra field do not match
-                AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED:"
-                    " The local file header contains file name which does not match the file name of the global file header."
-                    " The archive content is misconsistent with its directory. Please repair the archive");
-                return;
-            }
-
-            fileEntry.nFileDataOffset = pFileHeader->lLocalHeaderOffset + sizeof(ZipFile::LocalFileHeader) + pLocalFileHeader->nFileNameLength + pLocalFileHeader->nExtraFieldLength;
         }
-
-        // make sure it's the same file and the fileEntry structure is properly initialized
-        AZ_Assert(fileEntry.nFileHeaderOffset == pFileHeader->lLocalHeaderOffset, "The file entry header offset doesn't match the file header local offst");
-
-        fileEntry.nEOFOffset = fileEntry.nFileDataOffset + fileEntry.desc.lSizeCompressed;
-
-        if (fileEntry.nFileDataOffset >= m_nCDREndPos)
-        {
-            AZ_Warning("Archive", false, "ZD_ERROR_VALIDATION_FAILED:"
-                " The global file header declares the file which crosses the boundaries of the archive."
-                " The archive is either corrupted or truncated, please try to repair it");
-            return;
-        }
-
-        if (m_nInitMethod >= ZD_INIT_VALIDATE)
-        {
-            Validate(fileEntry);
-        }
-    }
-
-    //////////////////////////////////////////////////////////////////////////
-    // reads the file pointed by the given header and entry (they must be coherent)
-    // and decompresses it; then calculates and validates its CRC32
-    void CacheFactory::Validate(const FileEntryBase& fileEntry)
-    {
-        AZStd::vector<char> pBuffer;
-        // validate the file contents
-        // allocate memory for both the compressed data and uncompressed data
-        pBuffer.resize(fileEntry.desc.lSizeCompressed + fileEntry.desc.lSizeUncompressed);
-        char* pUncompressed = &pBuffer[fileEntry.desc.lSizeCompressed];
-        char* pCompressed = &pBuffer[0];
-
-        AZ_Assert(fileEntry.nFileDataOffset != FileEntry::INVALID_DATA_OFFSET, "File entry has invalid data offset of %" PRIx32, FileEntry::INVALID_DATA_OFFSET);
-        Seek(fileEntry.nFileDataOffset);
-
-        Read(pCompressed, fileEntry.desc.lSizeCompressed);
-
-        size_t nDestSize = fileEntry.desc.lSizeUncompressed;
-        int nError = Z_OK;
-        if (fileEntry.nMethod)
-        {
-            nError = ZipRawUncompress(pUncompressed, &nDestSize, pCompressed, fileEntry.desc.lSizeCompressed);
-        }
-        else
-        {
-            AZ_Assert(fileEntry.desc.lSizeCompressed == fileEntry.desc.lSizeUncompressed, "Uncompressed file does not have the same commpressed %u and uncompressed file sizes %u",
-                fileEntry.desc.lSizeCompressed, fileEntry.desc.lSizeUncompressed);
-            memcpy(pUncompressed, pCompressed, fileEntry.desc.lSizeUncompressed);
-        }
-        switch (nError)
-        {
-        case Z_OK:
-            break;
-        case Z_MEM_ERROR:
-            AZ_Warning("Archive", false, "ZD_ERROR_ZLIB_NO_MEMORY: ZLib reported out-of-memory error");
-            return;
-        case Z_BUF_ERROR:
-            AZ_Warning("Archive", false, "ZD_ERROR_ZLIB_CORRUPTED_DATA: ZLib reported compressed stream buffer error");
-            return;
-        case Z_DATA_ERROR:
-            AZ_Warning("Archive", false, "ZD_ERROR_ZLIB_CORRUPTED_DATA: ZLib reported compressed stream data error");
-            return;
-        default:
-            AZ_Warning("Archive", false, "ZD_ERROR_ZLIB_FAILED: ZLib reported an unexpected unknown error");
-            return;
-        }
-
-        if (nDestSize != fileEntry.desc.lSizeUncompressed)
-        {
-            AZ_Warning("Archive", false, "ZD_ERROR_CORRUPTED_DATA: Uncompressed stream doesn't match the size of uncompressed file stored in the archive file headers");
-            return;
-        }
-
-        uLong uCRC32 = AZ::Crc32((Bytef*)pUncompressed, nDestSize);
-        if (uCRC32 != fileEntry.desc.lCRC32)
-        {
-            AZ_Warning("Archive", false, "ZD_ERROR_CRC32_CHECK: Uncompressed stream CRC32 check failed");
-            return;
-        }
-    }
-
-
-    //////////////////////////////////////////////////////////////////////////
-    // extracts the file path from the file header with subsequent information
-    // may, or may not, put all letters to lower-case (depending on whether the system is to be case-sensitive or not)
-    // it's the responsibility of the caller to ensure that the file name is in readable valid memory
-    char* CacheFactory::GetFilePath(const char* pFileName, uint16_t nFileNameLength)
-    {
-        static char strResult[AZ_MAX_PATH_LEN];
-        AZ_Assert(nFileNameLength < AZ_MAX_PATH_LEN, "Only filenames shorter than %zu can be copied from filename parameter", AZ_MAX_PATH_LEN);
-        memcpy(strResult, pFileName, nFileNameLength);
-        strResult[nFileNameLength] = 0;
-        for (int i = 0; i < nFileNameLength; i++)
-        {
-            strResult[i] = std::tolower(strResult[i], std::locale{});
-        }
-
-        return strResult;
     }
 
     // seeks in the file relative to the starting position

Code/Framework/AzFramework/AzFramework/Archive/ZipDirCacheFactory.h

@@ -39,7 +39,7 @@ namespace AZ::IO::ZipDir
 
         // initializes the internal structures
         // nFlags can have FLAGS_READ_ONLY flag, in this case the object will be opened only for reading
-        CacheFactory(InitMethodEnum nInitMethod, uint32_t nFlags = 0);
+        CacheFactory(InitMethod nInitMethod, uint32_t nFlags = 0);
         ~CacheFactory();
 
         // the new function creates a new cache
@@ -66,28 +66,6 @@ namespace AZ::IO::ZipDir
         // This function can actually modify strFilePath variable, make sure you use a copy of the real path.
         void AddFileEntry(char* strFilePath, const ZipFile::CDRFileHeader* pFileHeader, const SExtraZipFileData& extra);// throw (ErrorEnum);
 
-        // extracts the file path from the file header with subsequent information
-        // may, or may not, put all letters to lower-case (depending on whether the system is to be case-sensitive or not)
-        // it's the responsibility of the caller to ensure that the file name is in readable valid memory
-        char* GetFilePath(const ZipFile::CDRFileHeader* pFileHeader)
-        {
-            return GetFilePath((const char*)(pFileHeader + 1), pFileHeader->nFileNameLength);
-        }
-        // extracts the file path from the file header with subsequent information
-        // may, or may not, put all letters to lower-case (depending on whether the system is to be case-sensitive or not)
-        // it's the responsibility of the caller to ensure that the file name is in readable valid memory
-        char* GetFilePath(const ZipFile::LocalFileHeader* pFileHeader)
-        {
-            return GetFilePath((const char*)(pFileHeader + 1), pFileHeader->nFileNameLength);
-        }
-        // extracts the file path from the file header with subsequent information
-        // may, or may not, put all letters to lower-case (depending on whether the system is to be case-sensitive or not)
-        // it's the responsibility of the caller to ensure that the file name is in readable valid memory
-        char* GetFilePath(const char* pFileName, uint16_t nFileNameLength);
-
-        // validates (if the init method has the corresponding value) the given file/header
-        void Validate(const FileEntryBase& fileEntry);
-
         // initializes the actual data offset in the file in the fileEntry structure
         // searches to the local file header, reads it and calculates the actual offset in the file
         void InitDataOffset(FileEntryBase& fileEntry, const ZipFile::CDRFileHeader* pFileHeader);
@@ -104,7 +82,7 @@ namespace AZ::IO::ZipDir
         AZStd::string m_szFilename;
         CZipFile m_fileExt;
 
-        InitMethodEnum m_nInitMethod;
+        InitMethod m_nInitMethod;
         uint32_t m_nFlags;
         ZipFile::CDREnd m_CDREnd;
 
@@ -129,7 +107,7 @@ namespace AZ::IO::ZipDir
         ZipFile::CryCustomEncryptionHeader m_headerEncryption;
         ZipFile::CrySignedCDRHeader m_headerSignature;
         ZipFile::CryCustomExtendedHeader m_headerExtended;
-
     };
+
 }
 

Code/Framework/AzFramework/AzFramework/Archive/ZipDirList.cpp

@@ -68,14 +68,14 @@ namespace AZ::IO::ZipDir
     {
         for (FileEntryTree::SubdirMap::iterator it = pTree->GetDirBegin(); it != pTree->GetDirEnd(); ++it)
         {
-            AddAllFiles(it->second.get(), (AZ::IO::Path(strRoot) / it->first).Native());
+            AddAllFiles(it->second.get(), (AZ::IO::Path(strRoot, AZ::IO::PosixPathSeparator) / it->first).Native());
         }
 
         for (FileEntryTree::FileMap::iterator it = pTree->GetFileBegin(); it != pTree->GetFileEnd(); ++it)
         {
             FileRecord rec;
             rec.pFileEntryBase = pTree->GetFileEntry(it);
-            rec.strPath = (AZ::IO::Path(strRoot) / it->first).Native();
+            rec.strPath = (AZ::IO::Path(strRoot, AZ::IO::PosixPathSeparator) / it->first).Native();
             push_back(rec);
         }
     }

Code/Framework/AzFramework/AzFramework/Archive/ZipDirStructures.h

@@ -119,19 +119,28 @@ namespace AZ::IO::ZipDir
         const char* m_szDescription;
     };
 
+#if defined(_RELEASE)
+    inline static constexpr bool IsReleaseConfig{ true };
+#else
+    inline static constexpr bool IsReleaseConfig{};
+#endif // _RELEASE
+
     // possible initialization methods
-    enum InitMethodEnum
+    enum class InitMethod
     {
-        // initialize as fast as possible, with minimal validation
-        ZD_INIT_FAST,
-        // after initialization, scan through all file headers, precache the actual file data offset values and validate the headers
-        ZD_INIT_FULL,
-        // scan all file headers and try to decompress the data, searching for corrupted files
-        ZD_INIT_VALIDATE_IN_MEMORY,
-        // store archive in memory
-        ZD_INIT_VALIDATE,
-        // maximum level of validation, checks for integrity of the archive
-        ZD_INIT_VALIDATE_MAX = ZD_INIT_VALIDATE
+        // initializes without any sort of extra validation steps
+        Default,
+
+        // initializes with extra validation steps
+        // not available in RELEASE
+        // will check CDR and local headers data match
+        ValidateHeaders,
+
+        // initializes with extra validation steps
+        // not available in RELEASE
+        // will check CDR and local headers data match
+        // will check file data CRC matches (when file is read)
+        FullValidation,
     };
 
     // Uncompresses raw (without wrapping) data that is compressed with method 8 (deflated) in the Zip file
@@ -184,7 +193,11 @@ namespace AZ::IO::ZipDir
         // the offset to the start of the next file's header - this
         // can be used to calculate the available space in zip file
         uint32_t nEOFOffset{};
+
+        // whether to check the CRC upon the next data read
+        bool bCheckCRCNextRead{};
     };
+
     // this is the record about the file in the Zip file.
     struct FileEntry
         : FileEntryBase

Code/Framework/AzFramework/AzFramework/Network/AssetProcessorConnection.cpp

@@ -593,7 +593,7 @@ namespace AzFramework
             
             DebugMessage("StartThread: Starting %s", thread.m_desc.m_name);
             thread.m_join = false;
-            thread.m_thread = AZStd::thread(thread.m_main, &thread.m_desc);
+            thread.m_thread = AZStd::thread(thread.m_desc, thread.m_main);
         }
 
         void AssetProcessorConnection::JoinThread(ThreadState& thread, AZStd::condition_variable* wakeUpCondition /* = nullptr */)

Code/Framework/AzFramework/AzFramework/TargetManagement/TargetManagementComponent.cpp

@@ -319,7 +319,7 @@ namespace AzFramework
         AZStd::thread_desc td;
         td.m_name = "TargetManager Thread";
         td.m_cpuId = AFFINITY_MASK_USERTHREADS;
-        m_threadHandle = AZStd::thread(AZStd::bind(&TargetManagementComponent::TickThread, this), &td);
+        m_threadHandle = AZStd::thread(td, AZStd::bind(&TargetManagementComponent::TickThread, this));
     }
 
     void TargetManagementComponent::Deactivate()