birkeh/o3de
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Archive Component - Rewrite and additional work on Archive and Asset Bundler (#4332)

Browse Source 
* Fix issues with seedlist for AutomatedTesting

Fixes error reporting so it will show the file hint in the tool.
Removes any missing assets from the .seed file.
Remove an unnecessary dependency from AutomatedTesting dependencies
file.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Make ArchiveComponent use AZ::IO::IArchive

Initial changes that will get the sychronous calls in ArchiveComponent
to use IArchive interface rather than external zip/7z tools.

Some of the asynchronous api are still in place, anything that wasn't
being used has been removed for now.  This may change later if we move
towards all the api being asynchronous.  Until then, we can't remove the
reliance upon the external archive tools completely.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Updates AZStd::thread constructors

Adds a variadic constructor which forwards args to the functor.
Because of our thread_desc extension, there was confusion on the
arugments, so the args were reordered to take the thread_desc first,
before the functor and args.
Also the thread_desc is taken as reference rather than by pointer.
Update callsites to account for this change.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Async operation of ArchiveComponent api

This sets up the ArchiveComponent to operate asynchronously.  It uses
promise/future to transfer results to caller.

This is still broken, there's a few things that need to get fixed up,
but this is a good checkpoint for the work as it solidifies the api,
cleans up a bunch of unused code, and compiles.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Removes the platform-specific ArchiveComponen

These are no longer needed, as they control the direct interaction with
host OS tools like 7za.exe or /bin/zip.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Removes the platform-specific files from cmake

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Removes the 7za.exe (and legal notice)

This tool is no longer needed in the repo.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fixes usage of IArchive::GetFullPath()

This changed to return a PathView, updated to reflect that.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix promises and threads

Make sure promises are only set exactly once.  This meant reworking some
of the initial error checking.
Detach threads when created.  Adds [[nodiscard]] to the functions that
return a future.  Since threads are detached, the future is the main way
to get communication from the thread.
Clean up interface, add comments.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* More edits to thread creation

Changes to thread construction to account for parameter change.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix some remaining issues with ArchiveComponent

Put created threads inside a container, then join them at Deactivate.
Fix asset bundler case when injecting a file with no working directory.
Fix thread constructor that applies args to a function.
Fix lambdas to take string args by value rather than reference.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fixes some remaining bugs in ArchiveComponent

Open archive as read-only during extract & list operations.
Fixes paths issues.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix initialize of opaque thread handle in thread_UnixLike.h

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>

* Removed unused variable in AssetBundleComponent.cpp to fix compiler warning

Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>

* Fix some issues with archives

File paths in the CDR and the local headers need to match, but there
were issues with path separators and case that made it possible to get
invalid headers errors in some archives.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Adds some new ArchiveComponent unit tests

Adds new tests for extraction of archive and adding files from a file
list to an archive.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Fix file data offset issues when opening archives

When opening an INestedArchive it would run through the CDR headers to
create file entries in the zip cache.  The offsets to the compressed
data were being calculated incorrectly because they were using the CDR
headers rather than jumping to the local file headers and getting
offsets from those sizes.

Removed and refactored some archive validation flow and zip cache
factory init methods to either init default or init w/ additional
validation checks.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Addresses PR feedback

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address more points of feedback in PR

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address additional PR feedback

 Fixes up some error checks and uses of strings vs paths.
 Enable archive component tests on Linux so we can see if they will
 pass.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address PR feedback

Change the INestedArchive interface to list files as AZ::IO::Path.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Disabling the ArchiveComponent tests on Linux

They failed so we will revisit them to attempt a fix.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Rename a member variable to be more accurate

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Address feedback on PR

Bump version of Archive Components for serialize context.
Improve error messages during archive open and validation.

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

* Revert recent changes

Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>

Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
This commit is contained in:
amzn-phist
2021-10-01 10:15:16 -05:00
committed by GitHub
parent b1fd428a99
commit fec77632ef
57 changed files with 1208 additions and 2600 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Code/Framework/AzToolsFramework/AzToolsFramework/Archive/ArchiveAPI.h
+56 -67
View File
@@ -12,95 +12,84 @@
#include <AzCore/EBus/EBus.h>
#include <AzCore/std/string/string.h>
#include <AzCore/Math/Uuid.h>
#include <future>
namespace AzToolsFramework
{
// use bind if you need additional context.
// Parameters:
// bool - If the archive command was successful or not.
typedef AZStd::function<void(bool)> ArchiveResponseCallback;
// bool - If the archive command was successful or not.
// AZStd::string - The console output from the command.
typedef AZStd::function<void(bool, AZStd::string)> ArchiveResponseOutputCallback;
//! ArchiveCommands
//! This bus handles messages relating to archive commands
//! archive commands are ASYNCHRONOUS
//! archive formats officially supported are .zip
//! do not block the main thread waiting for a response, it is not okay
//! you will not get a message delivered unless you tick the tickbus anyway!
class ArchiveCommands
: public AZ::EBusTraits
{
public:
using Bus = AZ::EBus<ArchiveCommands>;
// EBus Traits
static const AZ::EBusHandlerPolicy HandlerPolicy = AZ::EBusHandlerPolicy::Single;
static const AZ::EBusAddressPolicy AddressPolicy = AZ::EBusAddressPolicy::Single;
typedef AZStd::recursive_mutex MutexType;
static const bool LocklessDispatch = true;
virtual ~ArchiveCommands() {}
//! Start an async task to extract an archive to the target directory
//! taskHandles are used to cancel a task at some point in the future and are provided by the caller per task.
//! Multiple tasks can be associated with the same handle
virtual void ExtractArchive(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseCallback& respCallback) = 0;
virtual void ExtractArchiveOutput(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
// Maintaining backwards API compatibility - ExtractArchiveBlocking below passes in extractWithRoot as an option
virtual void ExtractArchiveWithoutRoot(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
virtual ~ArchiveCommands() = default;
//! Start a sync task to extract an archive to the target directory
//! If you do not want to extract the root folder then set extractWithRootDirectory to false.
virtual bool ExtractArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool extractWithRootDirectory) = 0;
//! Create an archive of the target directory (all files and subdirectories)
//! @param archivePath The path of the archive to create
//! @dirToArchive The directory to be added to the archive
//! @return Future (bool) which can obtain the success value of the operation
[[nodiscard]] virtual std::future<bool> CreateArchive(
const AZStd::string& archivePath,
const AZStd::string& dirToArchive) = 0;
//! Extract a single file asynchronously from the archive to the destination.
//! Uses cwd if destinationPath empty. overWrite = true for overwrite existing files, false for skipExisting
//! taskHandles are used to cancel a task at some point in the future and are provided by the caller per task.
//! Multiple tasks can be associated with the same handle
virtual void ExtractFile(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
//! Extract an archive to the target directory
//! @param archivePath The path of the archive to extract
//! @param destinationPath The directory where files will be extracted to
//! @return Future (bool) which can obtain the success value of the operation
[[nodiscard]] virtual std::future<bool> ExtractArchive(
const AZStd::string& archivePath,
const AZStd::string& destinationPath) = 0;
//! Extract a single file from the archive to the destination and block until finished.
//! Uses cwd if destinationPath empty. overWrite = true for overwrite existing files, false for skipExisting
virtual bool ExtractFileBlocking(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite) = 0;
//! Extract a single file from the archive to the destination
//! Destination path should not be empty
//! @param archivePath The path of the archive to extract from
//! @param fileInArchive A path to a file, relative to root of archive
//! @param destinationPath The directory where file will be extracted to
//! @return Future (bool) which can obtain the success value of the operation
[[nodiscard]] virtual std::future<bool> ExtractFile(
const AZStd::string& archivePath,
const AZStd::string& fileInArchive,
const AZStd::string& destinationPath) = 0;
//! Start an async task to create an archive of the target directory (recursively)
//! taskHandles are used to cancel a task at some point in the future and are provided by the caller per task.
//! Multiple tasks can be associated with the same handle.
virtual void CreateArchive(const AZStd::string& archivePath, const AZStd::string& dirToArchive, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
//! Retrieve the list of files contained in an archive (all files and subdirectories)
//! @param archivePath The path of the archive to list
//! @param outFileEntries An out parameter that will contain the file paths found
//! @return True if successful, false otherwise
virtual bool ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& outFileEntries) = 0;
//! Start a sync task to create an archive of the target directory (recursively)
virtual bool CreateArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& dirToArchive) = 0;
//! Start an async task to retrieve the list of files and their relative paths within an archive (recursively)
//! taskHandles are used to cancel a task at some point in the future and are provided by the caller per task.
//! Multiple tasks can be associated with the same handle.
virtual void ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& fileEntries, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
//! Add a file to an archive
//! The archive might not exist yet
//! The file path relative to the working directory will be replicated in the archive
//! @param archivePath The path of the archive to add to
//! @param workingDirectory A directory that will be the starting path of the file to be added
//! @param fileToAdd A path to the file relative to the working directory
//! @return Future (bool) which can obtain the success value of the operation
[[nodiscard]] virtual std::future<bool> AddFileToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& fileToAdd) = 0;
//! Start a sync task to retrieve the list of files and their relative paths within an archive (recursively)
virtual bool ListFilesInArchiveBlocking(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& fileEntries) = 0;
//! Start an async task to add a file to a preexisting archive.
//! fileToAdd must be a relative path to the file from the working directory. The path to the file from the root of the archive will be the same as the relative path to the file on disk.
//! taskHandles are used to cancel a task at some point in the future and are provided by the caller per task.
//! Multiple tasks can be associated with the same handle.
virtual void AddFileToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& fileToAdd, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
//! Start a sync task to add a file to a preexisting archive.
//! fileToAdd must be a relative path to the file from the working directory. The path to the file from the root of the archive will be the same as the relative path to the file on disk.
virtual bool AddFileToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& fileToAdd) = 0;
//! Start an async task to add files to a archive.
//! File paths inside the list file must either be a relative path from the working directory or an absolute path.
virtual void AddFilesToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) = 0;
//! Start a sync task to add files to an archive.
//! File paths inside the list file must either be a relative path from the working directory or an absolute path.
virtual bool AddFilesToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath) = 0;
//! Cancels tasks associtated with the given handle. Blocks until all tasks are cancelled.
virtual void CancelTasks(AZ::Uuid taskHandle) = 0;
//! Add files to an archive provided from a file listing
//! The archive might not exist yet
//! File paths in the file list should be relative to root of the archive
//! @param archivePath The path of the archive to add to
//! @param workingDirectory A directory that will be the starting path of the list of files to add
//! @param listFilePath Full path to a text file that contains the list of files to add
//! @return Future (bool) which can obtain the success value of the operation
[[nodiscard]] virtual std::future<bool> AddFilesToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& listFilePath) = 0;
};
using ArchiveCommandsBus = AZ::EBus<ArchiveCommands>;
}; // namespace AzToolsFramework
Code/Framework/AzToolsFramework/AzToolsFramework/Archive/ArchiveComponent.cpp
+507 -367
View File
@@ -12,118 +12,98 @@
#include <AzCore/Component/ComponentApplicationBus.h>
#include <AzCore/Serialization/EditContext.h>
#include <AzFramework/StringFunc/StringFunc.h>
#include <AzFramework/Archive/INestedArchive.h>
#include <AzFramework/Archive/ZipDirStructures.h>
#include <AzFramework/Process/ProcessCommunicator.h>
#include <AzFramework/Process/ProcessWatcher.h>
#include <AzFramework/FileFunc/FileFunc.h>
namespace AzToolsFramework
{
// Forward declare platform specific functions
namespace Platform
constexpr const char s_traceName[] = "ArchiveComponent";
constexpr AZ::u32 s_compressionMethod = AZ::IO::INestedArchive::METHOD_DEFLATE;
constexpr AZ::s32 s_compressionLevel = AZ::IO::INestedArchive::LEVEL_NORMAL;
constexpr CompressionCodec::Codec s_compressionCodec = CompressionCodec::Codec::ZLIB;
namespace ArchiveUtils
{
AZStd::string GetZipExePath();
AZStd::string GetUnzipExePath();
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive);
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot);
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file);
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& listFilePath);
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite);
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath);
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries);
}
const char s_traceName[] = "ArchiveComponent";
const unsigned int g_sleepDuration = 1;
// Echoes all results of stdout and stderr to console and never blocks
class ConsoleEchoCommunicator
{
public:
ConsoleEchoCommunicator(AzFramework::ProcessCommunicator* communicator)
: m_communicator(communicator)
// Read a file's contents into a provided buffer.
// Does not add a zero byte at the end of the buffer.
// returns true if read was successful, false otherwise.
bool ReadFile(const AZ::IO::Path& filePath, AZ::IO::OpenMode openMode, AZStd::vector<char>& outBuffer)
{
}
~ConsoleEchoCommunicator()
{
}
// Call this periodically to drain the buffers
void Pump()
{
if (m_communicator->IsValid())
auto fileIO = AZ::IO::FileIOBase::GetDirectInstance();
if (!fileIO)
{
AZ::u32 readBufferSize = 0;
AZStd::string readBuffer;
// Don't call readOutput unless there is output or else it will block...
readBufferSize = m_communicator->PeekOutput();
if (readBufferSize)
{
readBuffer.resize_no_construct(readBufferSize + 1);
readBuffer[readBufferSize] = '\0';
m_communicator->ReadOutput(readBuffer.data(), readBufferSize);
EchoBuffer(readBuffer);
}
readBufferSize = m_communicator->PeekError();
if (readBufferSize)
{
readBuffer.resize_no_construct(readBufferSize + 1);
readBuffer[readBufferSize] = '\0';
m_communicator->ReadError(readBuffer.data(), readBufferSize);
EchoBuffer(readBuffer);
}
return false;
}
}
private:
void EchoBuffer(const AZStd::string& buffer)
{
size_t startIndex = 0;
size_t endIndex = 0;
const size_t bufferSize = buffer.size();
for (size_t i = 0; i < bufferSize; ++i)
bool success = false;
AZ::IO::HandleType fileHandle = AZ::IO::InvalidHandle;
if (fileIO->Open(filePath.c_str(), openMode, fileHandle))
{
if (buffer[i] == '\n' || buffer[i] == '\0')
AZ::u64 fileSize = 0;
if (fileIO->Size(fileHandle, fileSize) && fileSize != 0)
{
endIndex = i;
bool isEmptyMessage = (endIndex - startIndex == 1) && (buffer[startIndex] == '\r');
if (!isEmptyMessage)
outBuffer.resize_no_construct(fileSize);
AZ::u64 bytesRead = 0;
if (fileIO->Read(fileHandle, outBuffer.data(), fileSize, true, &bytesRead))
{
AZ_Printf(s_traceName, "%s", buffer.substr(startIndex, endIndex - startIndex).c_str());
success = (fileSize == bytesRead);
}
startIndex = endIndex + 1;
}
fileIO->Close(fileHandle);
}
return success;
}
// Reads a text file that contains a list of file paths.
// Tokenize the file by lines.
// Calls the lineVisitor function for each line of the file.
void ProcessFileList(const AZ::IO::Path& filePath, AZStd::function<void(AZStd::string_view line)> lineVisitor)
{
AZStd::vector<char> fileBuffer;
if (ReadFile(filePath, AZ::IO::OpenMode::ModeText | AZ::IO::OpenMode::ModeRead, fileBuffer))
{
AZ::StringFunc::TokenizeVisitor(AZStd::string_view{ fileBuffer.data(), fileBuffer.size() }, lineVisitor, "\n");
}
}
AzFramework::ProcessCommunicator* m_communicator = nullptr;
};
} // namespace ArchiveUtils
void ArchiveComponent::Activate()
{
m_zipExePath = Platform::GetZipExePath();
m_unzipExePath = Platform::GetUnzipExePath();
m_fileIO = AZ::IO::FileIOBase::GetDirectInstance();
if (m_fileIO == nullptr)
{
AZ_Error(s_traceName, false, "Failed to create a LocalFileIO instance!");
}
ArchiveCommands::Bus::Handler::BusConnect();
m_archive = AZ::Interface<AZ::IO::IArchive>::Get();
if (m_archive == nullptr)
{
AZ_Error(s_traceName, false, "Failed to get IArchive interface!");
}
ArchiveCommandsBus::Handler::BusConnect();
}
void ArchiveComponent::Deactivate()
{
ArchiveCommands::Bus::Handler::BusDisconnect();
ArchiveCommandsBus::Handler::BusDisconnect();
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
for (auto pair : m_threadInfoMap)
m_fileIO = nullptr;
m_archive = nullptr;
for (AZStd::thread& t : m_threads)
{
ThreadInfo& info = pair.second;
info.shouldStop = true;
m_cv.wait(lock, [&info]() {
return info.threads.size() == 0;
});
t.join();
}
m_threadInfoMap.clear();
m_threads = {};
}
void ArchiveComponent::Reflect(AZ::ReflectContext * context)
@@ -132,7 +112,7 @@ namespace AzToolsFramework
{
serializeContext->Class<ArchiveComponent, AZ::Component>()
->Version(2)
->Attribute(AZ::Edit::Attributes::SystemComponentTags, AZStd::vector<AZ::Crc32>({ AZ_CRC("AssetBuilder", 0xc739c7d7) }))
->Attribute(AZ::Edit::Attributes::SystemComponentTags, AZStd::vector<AZ::Crc32>({ AZ_CRC_CE("AssetBuilder") }))
;
if (AZ::EditContext* editContext = serializeContext->GetEditContext())
@@ -141,320 +121,480 @@ namespace AzToolsFramework
"Archive", "Handles creation and extraction of zip archives.")
->ClassElement(AZ::Edit::ClassElements::EditorData, "")
->Attribute(AZ::Edit::Attributes::Category, "Editor")
->Attribute(AZ::Edit::Attributes::AppearsInAddComponentMenu, AZ_CRC("System", 0xc94d118b))
->Attribute(AZ::Edit::Attributes::AppearsInAddComponentMenu, AZ_CRC_CE("System"))
;
}
}
}
void ArchiveComponent::CreateArchive(const AZStd::string& archivePath, const AZStd::string& dirToArchive, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
std::future<bool> ArchiveComponent::CreateArchive(
const AZStd::string& archivePath,
const AZStd::string& dirToArchive)
{
AZStd::string commandLineArgs = AZStd::string::format(R"(a -tzip -mx=1 "%s" -r "%s\*")", archivePath.c_str(), dirToArchive.c_str());
LaunchZipExe(m_zipExePath, commandLineArgs, respCallback, taskHandle);
}
bool ArchiveComponent::CreateArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
{
bool success = false;
auto createArchiveCallback = [&success](bool result, AZStd::string consoleOutput) {
success = result;
};
AZStd::string commandLineArgs = Platform::GetCreateArchiveCommand(archivePath, dirToArchive);
if (commandLineArgs.empty())
if (!CheckParamsForCreate(archivePath, dirToArchive))
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return false;
std::promise<bool> p;
p.set_value(false);
return p.get_future();
}
LaunchZipExe(m_zipExePath, commandLineArgs, createArchiveCallback, AZ::Uuid::CreateNull(), dirToArchive, false);
return success;
}
void ArchiveComponent::ExtractArchive(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseCallback& respCallback)
{
ArchiveResponseOutputCallback responseHandler = [respCallback](bool result, AZStd::string /*outputStr*/) { respCallback(result); };
ExtractArchiveOutput(archivePath, destinationPath, taskHandle, responseHandler);
}
void ArchiveComponent::ExtractArchiveOutput(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
{
AZStd::string commandLineArgs = Platform::GetExtractArchiveCommand(archivePath, destinationPath, true);
if (commandLineArgs.empty())
auto FnCreateArchive = [this, archivePath, dirToArchive](std::promise<bool>&& p) -> void
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return;
}
LaunchZipExe(m_unzipExePath, commandLineArgs, respCallback, taskHandle);
}
void ArchiveComponent::ExtractArchiveWithoutRoot(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
{
AZStd::string commandLineArgs = Platform::GetExtractArchiveCommand(archivePath, destinationPath, false);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return;
}
LaunchZipExe(m_unzipExePath, commandLineArgs, respCallback, taskHandle);
}
void ArchiveComponent::ExtractFile(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
{
AZStd::string commandLineArgs = AzToolsFramework::Platform::GetExtractFileCommand(archivePath, fileInArchive, destinationPath, overWrite);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return;
}
LaunchZipExe(m_unzipExePath, commandLineArgs, respCallback, taskHandle);
}
bool ArchiveComponent::ExtractFileBlocking(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
{
AZStd::string commandLineArgs = AzToolsFramework::Platform::GetExtractFileCommand(archivePath, fileInArchive, destinationPath, overWrite);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return false;
}
bool success = false;
auto createArchiveCallback = [&success](bool result, AZStd::string consoleOutput) {
success = result;
};
LaunchZipExe(m_unzipExePath, commandLineArgs, createArchiveCallback);
return success;
}
void ArchiveComponent::ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& fileEntries, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
{
AZStd::string commandLineArgs = Platform::GetListFilesInArchiveCommand(archivePath);
auto parseOutput = [respCallback, &fileEntries](bool exitCode, AZStd::string consoleOutput)
{
Platform::ParseConsoleOutputFromListFilesInArchive(consoleOutput, fileEntries);
AZ::TickBus::QueueFunction(respCallback, exitCode, AZStd::move(consoleOutput));
};
LaunchZipExe(m_unzipExePath, commandLineArgs, parseOutput, taskHandle, "", true);
}
bool ArchiveComponent::ListFilesInArchiveBlocking(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& fileEntries)
{
AZStd::string listOutput;
AZStd::string commandLineArgs = Platform::GetListFilesInArchiveCommand(archivePath.c_str());
bool success = false;
auto parseOutput = [&success, &fileEntries](bool result, AZStd::string consoleOutput)
{
Platform::ParseConsoleOutputFromListFilesInArchive(consoleOutput, fileEntries);
success = result;
};
LaunchZipExe(m_unzipExePath, commandLineArgs, parseOutput, AZ::Uuid::CreateNull(), "", true);
return success;
}
void ArchiveComponent::AddFileToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& fileToAdd, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
{
AZStd::string commandLineArgs = Platform::GetAddFileToArchiveCommand(archivePath, fileToAdd);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return;
}
LaunchZipExe(m_zipExePath, commandLineArgs, respCallback, taskHandle, workingDirectory);
}
bool ArchiveComponent::AddFileToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& fileToAdd)
{
AZStd::string commandLineArgs = Platform::GetAddFileToArchiveCommand(archivePath, fileToAdd);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return false;
}
bool success = false;
auto addFileToArchiveCallback = [&success](bool result, AZStd::string consoleOutput) {
success = result;
};
LaunchZipExe(m_zipExePath, commandLineArgs, addFileToArchiveCallback, AZ::Uuid::CreateNull(), workingDirectory);
return success;
}
bool ArchiveComponent::AddFilesToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath)
{
bool success = false;
auto addFileToArchiveCallback = [&success](bool result, AZStd::string consoleOutput) {
success = result;
};
AZStd::string commandLineArgs = Platform::GetAddFilesToArchiveCommand(archivePath.c_str(), listFilePath.c_str());
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return false;
}
LaunchZipExe(m_zipExePath, commandLineArgs, addFileToArchiveCallback, AZ::Uuid::CreateNull(), workingDirectory);
return success;
}
void ArchiveComponent::AddFilesToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback)
{
AZStd::string commandLineArgs = Platform::GetAddFilesToArchiveCommand(archivePath, listFilePath);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return;
}
LaunchZipExe(m_zipExePath, commandLineArgs, respCallback, taskHandle, workingDirectory);
}
bool ArchiveComponent::ExtractArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool extractWithRootDirectory)
{
AZStd::string commandLineArgs = Platform::GetExtractArchiveCommand(archivePath, destinationPath, extractWithRootDirectory);
if (commandLineArgs.empty())
{
// The platform-specific implementation has already thrown its own error, no need to throw another one
return false;
}
bool success = false;
auto extractArchiveCallback = [&success](bool result, AZStd::string consoleOutput) {
success = result;
};
LaunchZipExe(m_unzipExePath, commandLineArgs, extractArchiveCallback);
return success;
}
void ArchiveComponent::CancelTasks(AZ::Uuid taskHandle)
{
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
auto it = m_threadInfoMap.find(taskHandle);
if (it == m_threadInfoMap.end())
{
return;
}
ThreadInfo& info = it->second;
info.shouldStop = true;
m_cv.wait(lock, [&info]() {
return info.threads.size() == 0;
});
m_threadInfoMap.erase(it);
}
void ArchiveComponent::LaunchZipExe(const AZStd::string& exePath, const AZStd::string& commandLineArgs, const ArchiveResponseOutputCallback& respCallback, AZ::Uuid taskHandle, const AZStd::string& workingDir, bool captureOutput)
{
auto sevenZJob = [=]()
{
if (!taskHandle.IsNull())
auto archive = m_archive->OpenArchive(archivePath, {}, AZ::IO::INestedArchive::FLAGS_CREATE_NEW);
if (!archive)
{
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
m_threadInfoMap[taskHandle].threads.insert(AZStd::this_thread::get_id());
m_cv.notify_all();
AZ_Error(s_traceName, false, "Failed to create archive file '%s'", archivePath.c_str());
p.set_value(false);
return;
}
AzFramework::ProcessLauncher::ProcessLaunchInfo info;
info.m_commandlineParameters = exePath + " " + commandLineArgs;
info.m_showWindow = false;
if (!workingDir.empty())
auto foundFiles = AzFramework::FileFunc::FindFilesInPath(dirToArchive, "*", true);
if (!foundFiles.IsSuccess())
{
info.m_workingDirectory = workingDir;
AZ_Error(s_traceName, false, "Failed to find file listing under directory '%d'", dirToArchive.c_str());
p.set_value(false);
return;
}
AZStd::unique_ptr<AzFramework::ProcessWatcher> watcher(AzFramework::ProcessWatcher::LaunchProcess(info, AzFramework::ProcessCommunicationType::COMMUNICATOR_TYPE_STDINOUT));
AZStd::string consoleOutput;
AZ::u32 exitCode = static_cast<AZ::u32>(SevenZipExitCode::UserStoppedProcess);
if (watcher)
bool success = true;
AZStd::vector<char> fileBuffer;
const AZ::IO::Path workingPath{ dirToArchive };
for (const auto& fileName : foundFiles.GetValue())
{
// callback requires output captured from 7z
if (captureOutput)
bool thisSuccess = false;
AZ::IO::PathView relativePath = AZ::IO::PathView{ fileName }.LexicallyRelative(workingPath);
AZ::IO::Path fullPath = (workingPath / relativePath);
if (ArchiveUtils::ReadFile(fullPath, AZ::IO::OpenMode::ModeRead, fileBuffer))
{
AZStd::string consoleBuffer;
while (watcher->IsProcessRunning(&exitCode))
{
if (!taskHandle.IsNull())
{
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
if (m_threadInfoMap[taskHandle].shouldStop)
{
watcher->TerminateProcess(static_cast<AZ::u32>(SevenZipExitCode::UserStoppedProcess));
}
}
watcher->WaitForProcessToExit(g_sleepDuration, &exitCode);
AZ::u32 outputSize = watcher->GetCommunicator()->PeekOutput();
if (outputSize)
{
consoleBuffer.resize(outputSize);
watcher->GetCommunicator()->ReadOutput(consoleBuffer.data(), outputSize);
consoleOutput += consoleBuffer;
}
}
int result = archive->UpdateFile(
relativePath.Native(), fileBuffer.data(), fileBuffer.size(), s_compressionMethod,
s_compressionLevel, s_compressionCodec);
thisSuccess = (result == AZ::IO::ZipDir::ZD_ERROR_SUCCESS);
AZ_Error(
s_traceName, thisSuccess, "Error %d encountered while adding '%s' to archive '%.*s'", result, fileName.c_str(),
AZ_STRING_ARG(archive->GetFullPath().Native()));
}
else
{
ConsoleEchoCommunicator echoCommunicator(watcher->GetCommunicator());
while (watcher->IsProcessRunning(&exitCode))
{
if (!taskHandle.IsNull())
{
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
if (m_threadInfoMap[taskHandle].shouldStop)
{
watcher->TerminateProcess(static_cast<AZ::u32>(SevenZipExitCode::UserStoppedProcess));
}
}
watcher->WaitForProcessToExit(g_sleepDuration, &exitCode);
echoCommunicator.Pump();
}
AZ_Error(
s_traceName, false, "Error encountered while reading '%s' to add to archive '%.*s'", fileName.c_str(),
AZ_STRING_ARG(archive->GetFullPath().Native()));
}
success = (success && thisSuccess);
}
if (taskHandle.IsNull())
archive.reset();
p.set_value(success);
};
// Async task...
std::promise<bool> p;
std::future<bool> f = p.get_future();
AZStd::thread_desc threadDesc;
threadDesc.m_name = "Archive Task (Create)";
m_threads.emplace_back(threadDesc, FnCreateArchive, AZStd::move(p));
return f;
}
std::future<bool> ArchiveComponent::ExtractArchive(
const AZStd::string& archivePath,
const AZStd::string& destinationPath)
{
if (!CheckParamsForExtract(archivePath, destinationPath))
{
std::promise<bool> p;
p.set_value(false);
return p.get_future();
}
auto FnExtractArchive = [this, archivePath, destinationPath](std::promise<bool>&& p) -> void
{
auto archive = m_archive->OpenArchive(archivePath, {}, AZ::IO::INestedArchive::FLAGS_READ_ONLY);
if (!archive)
{
respCallback(exitCode == static_cast<AZ::u32>(SevenZipExitCode::NoError), AZStd::move(consoleOutput));
AZ_Error(s_traceName, false, "Failed to open archive file '%s'", archivePath.c_str());
p.set_value(false);
return;
}
AZStd::vector<AZ::IO::Path> filesInArchive;
if (int result = archive->ListAllFiles(filesInArchive); result != AZ::IO::ZipDir::ZD_ERROR_SUCCESS)
{
AZ_Error(s_traceName, false, "Failed to get list of files in archive '%s'", archivePath.c_str());
p.set_value(false);
return;
}
AZStd::vector<AZ::u8> fileBuffer;
AZ::IO::Path destination{ destinationPath };
AZ::u64 fileSize = 0;
AZ::u64 numFilesWritten = 0;
AZ::u64 bytesWritten = 0;
AZ::IO::INestedArchive::Handle srcHandle{};
AZ::IO::HandleType dstHandle = AZ::IO::InvalidHandle;
constexpr AZ::IO::OpenMode openMode =
(AZ::IO::OpenMode::ModeCreatePath | AZ::IO::OpenMode::ModeWrite | AZ::IO::OpenMode::ModeUpdate);
for (const auto& filePath : filesInArchive)
{
srcHandle = archive->FindFile(filePath.Native());
AZ_Assert(srcHandle != nullptr, "File '%s' does not exist inside archive '%s'", filePath.c_str(), archivePath.c_str());
fileSize = (srcHandle != nullptr) ? archive->GetFileSize(srcHandle) : 0;
fileBuffer.resize_no_construct(fileSize);
if (auto result = archive->ReadFile(srcHandle, fileBuffer.data()); result != AZ::IO::ZipDir::ZD_ERROR_SUCCESS)
{
AZ_Error(
s_traceName, false, "Failed to read file '%s' in archive '%s' with error %d", filePath.c_str(), archivePath.c_str(),
result);
continue;
}
AZ::IO::Path destinationFile = destination / filePath;
if (!m_fileIO->Open(destinationFile.c_str(), openMode, dstHandle))
{
AZ_Error(s_traceName, false, "Failed to open '%s' for writing", destinationFile.c_str());
continue;
}
if (!m_fileIO->Write(dstHandle, fileBuffer.data(), fileSize, &bytesWritten))
{
AZ_Error(s_traceName, false, "Failed to write destination file '%s'", destinationFile.c_str());
}
else if (bytesWritten == fileSize)
{
++numFilesWritten;
}
m_fileIO->Close(dstHandle);
}
p.set_value(numFilesWritten == filesInArchive.size());
};
// Async task...
std::promise<bool> p;
std::future<bool> f = p.get_future();
AZStd::thread_desc threadDesc;
threadDesc.m_name = "Archive Task (Extract)";
m_threads.emplace_back(threadDesc, FnExtractArchive, AZStd::move(p));
return f;
}
std::future<bool> ArchiveComponent::ExtractFile(
const AZStd::string& archivePath,
const AZStd::string& fileInArchive,
const AZStd::string& destinationPath)
{
if (!CheckParamsForExtract(archivePath, destinationPath))
{
std::promise<bool> p;
p.set_value(false);
return p.get_future();
}
auto FnExtractFile = [this, archivePath, fileInArchive, destinationPath](std::promise<bool>&& p) -> void
{
auto archive = m_archive->OpenArchive(archivePath, {}, AZ::IO::INestedArchive::FLAGS_READ_ONLY);
if (!archive)
{
AZ_Error(s_traceName, false, "Failed to open archive file '%s'", archivePath.c_str());
p.set_value(false);
return;
}
AZ::IO::INestedArchive::Handle fileHandle = archive->FindFile(fileInArchive);
if (!fileHandle)
{
AZ_Error(s_traceName, false, "File '%s' does not exist inside archive '%s'", fileInArchive.c_str(), archivePath.c_str());
p.set_value(false);
return;
}
AZ::u64 fileSize = archive->GetFileSize(fileHandle);
AZStd::vector<AZ::u8> fileBuffer;
fileBuffer.resize_no_construct(fileSize);
if (auto result = archive->ReadFile(fileHandle, fileBuffer.data()); result != AZ::IO::ZipDir::ZD_ERROR_SUCCESS)
{
AZ_Error(
s_traceName, false, "Failed to read file '%s' in archive '%s' with error %d", fileInArchive.c_str(),
archivePath.c_str(), result);
p.set_value(false);
return;
}
AZ::IO::HandleType destFileHandle = AZ::IO::InvalidHandle;
AZ::IO::Path destinationFile{ destinationPath };
destinationFile /= fileInArchive;
AZ::IO::OpenMode openMode = (AZ::IO::OpenMode::ModeCreatePath | AZ::IO::OpenMode::ModeWrite | AZ::IO::OpenMode::ModeUpdate);
if (!m_fileIO->Open(destinationFile.c_str(), openMode, destFileHandle))
{
AZ_Error(s_traceName, false, "Failed to open destination file '%s' for writing", destinationFile.c_str());
p.set_value(false);
return;
}
AZ::u64 bytesWritten = 0;
if (!m_fileIO->Write(destFileHandle, fileBuffer.data(), fileSize, &bytesWritten))
{
AZ_Error(s_traceName, false, "Failed to write destination file '%s'", destinationFile.c_str());
}
m_fileIO->Close(destFileHandle);
p.set_value(bytesWritten == fileSize);
};
// Async task...
std::promise<bool> p;
std::future<bool> f = p.get_future();
AZStd::thread_desc threadDesc;
threadDesc.m_name = "Archive Task (Extract Single)";
m_threads.emplace_back(threadDesc, FnExtractFile, AZStd::move(p));
return f;
}
bool ArchiveComponent::ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& outFileEntries)
{
if (!m_fileIO || !m_archive)
{
return false;
}
if (!m_fileIO->Exists(archivePath.c_str()))
{
AZ_Error(s_traceName, false, "Archive '%s' does not exist!", archivePath.c_str());
return false;
}
auto archive = m_archive->OpenArchive(archivePath, {}, AZ::IO::INestedArchive::FLAGS_READ_ONLY);
if (!archive)
{
AZ_Error(s_traceName, false, "Failed to open archive file '%s'", archivePath.c_str());
return false;
}
AZStd::vector<AZ::IO::Path> fileEntries;
int result = archive->ListAllFiles(fileEntries);
outFileEntries.clear();
for (const auto& path : fileEntries)
{
outFileEntries.emplace_back(path.String());
}
return (result == AZ::IO::ZipDir::ZD_ERROR_SUCCESS);
}
std::future<bool> ArchiveComponent::AddFileToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& fileToAdd)
{
if (!CheckParamsForAdd(workingDirectory, fileToAdd))
{
std::promise<bool> p;
p.set_value(false);
return p.get_future();
}
auto FnAddFileToArchive = [this, archivePath, workingDirectory, fileToAdd](std::promise<bool>&& p) -> void
{
auto archive = m_archive->OpenArchive(archivePath);
if (!archive)
{
AZ_Error(s_traceName, false, "Failed to open archive file '%s'", archivePath.c_str());
p.set_value(false);
return;
}
AZ::IO::Path workingPath{ workingDirectory };
AZ::IO::Path fullPath = workingPath / fileToAdd;
AZ::IO::PathView relativePath = AZ::IO::PathView{ fullPath }.LexicallyRelative(workingPath);
AZStd::vector<char> fileBuffer;
bool success = false;
if (ArchiveUtils::ReadFile(fullPath, AZ::IO::OpenMode::ModeRead, fileBuffer))
{
int result = archive->UpdateFile(
relativePath.Native(), fileBuffer.data(), fileBuffer.size(), s_compressionMethod,
s_compressionLevel, s_compressionCodec);
success = (result == AZ::IO::ZipDir::ZD_ERROR_SUCCESS);
AZ_Error(
s_traceName, success, "Error %d encountered while adding '%s' to archive '%.*s'", result, fileToAdd.c_str(),
AZ_STRING_ARG(archive->GetFullPath().Native()));
}
else
{
AZ::TickBus::QueueFunction(respCallback, (exitCode == static_cast<AZ::u32>(SevenZipExitCode::NoError)), AZStd::move(consoleOutput));
AZ_Error(
s_traceName, false, "Error encountered while reading '%s' to add to archive '%.*s'", fileToAdd.c_str(),
AZ_STRING_ARG(archive->GetFullPath().Native()));
}
if (!taskHandle.IsNull())
{
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
ThreadInfo& tInfo = m_threadInfoMap[taskHandle];
tInfo.threads.erase(AZStd::this_thread::get_id());
m_cv.notify_all();
}
archive.reset();
p.set_value(success);
};
if (!taskHandle.IsNull())
{
AZStd::thread processThread(sevenZJob);
AZStd::unique_lock<AZStd::mutex> lock(m_threadControlMutex);
ThreadInfo& info = m_threadInfoMap[taskHandle];
m_cv.wait(lock, [&info, &processThread]() {
return info.threads.find(processThread.get_id()) != info.threads.end();
});
processThread.detach();
}
else
{
sevenZJob();
}
// Async task...
std::promise<bool> p;
std::future<bool> f = p.get_future();
AZStd::thread_desc threadDesc;
threadDesc.m_name = "Archive Task (Add Single)";
m_threads.emplace_back(threadDesc, FnAddFileToArchive, AZStd::move(p));
return f;
}
std::future<bool> ArchiveComponent::AddFilesToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& listFilePath)
{
if (!CheckParamsForAdd(workingDirectory, listFilePath))
{
std::promise<bool> p;
p.set_value(false);
return p.get_future();
}
auto FnAddFilesToArchive = [this, archivePath, workingDirectory, listFilePath](std::promise<bool>&& p) -> void
{
auto archive = m_archive->OpenArchive(archivePath);
if (!archive)
{
AZ_Error(s_traceName, false, "Failed to open archive file '%s'", archivePath.c_str());
p.set_value(false);
return;
}
bool success = true; // starts true and turns false when any error is encountered.
AZ::IO::Path basePath{ workingDirectory };
auto PerLineCallback = [&success, &basePath, &archive](AZStd::string_view filePathLine) -> void
{
AZStd::vector<char> fileBuffer;
AZ::IO::Path fullPath = (basePath / filePathLine);
if (ArchiveUtils::ReadFile(fullPath, AZ::IO::OpenMode::ModeRead, fileBuffer))
{
int result = archive->UpdateFile(
filePathLine, fileBuffer.data(), fileBuffer.size(), s_compressionMethod,
s_compressionLevel, s_compressionCodec);
bool thisSuccess = (result == AZ::IO::ZipDir::ZD_ERROR_SUCCESS);
success = (success && thisSuccess);
AZ_Error(
s_traceName, thisSuccess, "Error %d encountered while adding '%.*s' to archive '%.*s'", result,
AZ_STRING_ARG(filePathLine), AZ_STRING_ARG(archive->GetFullPath().Native()));
}
else
{
AZ_Error(
s_traceName, false, "Error encountered while reading '%.*s' to add to archive '%.*s'", AZ_STRING_ARG(filePathLine),
AZ_STRING_ARG(archive->GetFullPath().Native()));
}
};
ArchiveUtils::ProcessFileList(listFilePath, PerLineCallback);
archive.reset();
p.set_value(success);
};
// Async task...
std::promise<bool> p;
std::future<bool> f = p.get_future();
AZStd::thread_desc threadDesc;
threadDesc.m_name = "Archive Task (Add)";
m_threads.emplace_back(threadDesc, FnAddFilesToArchive, AZStd::move(p));
return f;
}
bool ArchiveComponent::CheckParamsForAdd(const AZStd::string& directory, const AZStd::string& file)
{
if (!m_fileIO || !m_archive)
{
return false;
}
if (!m_fileIO->IsDirectory(directory.c_str()))
{
AZ_Error(
s_traceName, false, "Working directory '%s' is not a directory or doesn't exist!", directory.c_str());
return false;
}
if (!file.empty())
{
auto filePath = AZ::IO::Path{ directory } / file;
if (!m_fileIO->Exists(filePath.c_str()) || m_fileIO->IsDirectory(filePath.c_str()))
{
AZ_Error(s_traceName, false, "File list '%s' is a directory or doesn't exist!", filePath.c_str());
return false;
}
}
return true;
}
bool ArchiveComponent::CheckParamsForExtract(const AZStd::string& archive, const AZStd::string& directory)
{
if (!m_fileIO || !m_archive)
{
return false;
}
if (!m_fileIO->Exists(archive.c_str()))
{
AZ_Error(s_traceName, false, "Archive '%s' does not exist!", archive.c_str());
return false;
}
if (!m_fileIO->Exists(directory.c_str()))
{
if (!m_fileIO->CreatePath(directory.c_str()))
{
AZ_Error(s_traceName, false, "Failed to create destination directory '%s'", directory.c_str());
return false;
}
}
return true;
}
bool ArchiveComponent::CheckParamsForCreate(const AZStd::string& archive, const AZStd::string& directory)
{
if (!m_fileIO || !m_archive)
{
return false;
}
if (m_fileIO->Exists(archive.c_str()))
{
AZ_Error(s_traceName, false, "Archive file '%s' already exists, cannot create a new archive there!");
return false;
}
if (!m_fileIO->IsDirectory(directory.c_str()))
{
AZ_Error(s_traceName, false, "Directory '%s' is not a directory or doesn't exist!", directory.c_str());
return false;
}
return true;
}
} // namespace AzToolsFramework
Code/Framework/AzToolsFramework/AzToolsFramework/Archive/ArchiveComponent.h
+43 -47
View File
@@ -10,80 +10,76 @@
#include <AzCore/base.h>
#include <AzCore/Component/Component.h>
#include <AzCore/IO/FileIO.h>
#include <AzCore/std/parallel/thread.h>
#include <AzCore/std/parallel/conditional_variable.h>
#include <AzCore/std/containers/set.h>
#include <AzCore/std/containers/unordered_set.h>
#include <AzFramework/Archive/IArchive.h>
#include <AzToolsFramework/Archive/ArchiveAPI.h>
namespace AzToolsFramework
{
enum class SevenZipExitCode : AZ::u32
{
NoError = 0,
Warning = 1,
FatalError = 2,
CommandLineError = 7,
NotEnoughMemory = 8,
UserStoppedProcess = 255
};
// the ArchiveComponent's job is to execute zip commands.
// it parses the status of zip commands and returns results.
// the ArchiveComponent's job is to create and manipulate zip archives.
class ArchiveComponent
: public AZ::Component
, private ArchiveCommands::Bus::Handler
, private ArchiveCommandsBus::Handler
{
public:
AZ_COMPONENT(ArchiveComponent, "{A19EEA33-3736-447F-ACF7-DAA4B6A179AA}")
AZ_COMPONENT(ArchiveComponent, "{A19EEA33-3736-447F-ACF7-DAA4B6A179AA}");
ArchiveComponent() = default;
~ArchiveComponent() override = default;
ArchiveComponent(const ArchiveComponent&) = delete;
ArchiveComponent& operator=(const ArchiveComponent&) = delete;
//////////////////////////////////////////////////////////////////////////
// AZ::Component overrides
void Activate() override;
void Deactivate() override;
//////////////////////////////////////////////////////////////////////////
private:
protected:
static void Reflect(AZ::ReflectContext* context);
//////////////////////////////////////////////////////////////////////////
// ArchiveCommands::Bus::Handler overrides
void CreateArchive(const AZStd::string& archivePath, const AZStd::string& dirToArchive, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool CreateArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& dirToArchive) override;
bool ExtractArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool extractWithRootDirectory) override;
void ExtractArchive(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseCallback& respCallback) override;
void ExtractArchiveOutput(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
void ExtractArchiveWithoutRoot(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
void ExtractFile(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool ExtractFileBlocking(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite) override;
void ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& fileEntries, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool ListFilesInArchiveBlocking(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& fileEntries) override;
void AddFileToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& fileToAdd, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool AddFileToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& fileToAdd) override;
bool AddFilesToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath) override;
void AddFilesToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
void CancelTasks(AZ::Uuid taskHandle) override;
// ArchiveCommandsBus::Handler overrides
[[nodiscard]] std::future<bool> CreateArchive(
const AZStd::string& archivePath,
const AZStd::string& dirToArchive) override;
[[nodiscard]] std::future<bool> ExtractArchive(
const AZStd::string& archivePath,
const AZStd::string& destinationPath) override;
[[nodiscard]] std::future<bool> ExtractFile(
const AZStd::string& archivePath,
const AZStd::string& fileInArchive,
const AZStd::string& destinationPath) override;
bool ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& outFileEntries) override;
[[nodiscard]] std::future<bool> AddFileToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& fileToAdd) override;
[[nodiscard]] std::future<bool> AddFilesToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& listFilePath) override;
//////////////////////////////////////////////////////////////////////////
// Launches the input zip exe as a background child process in a detached background thread, if the task handle is not null
// otherwise launches input zip exe in the calling thread.
void LaunchZipExe(const AZStd::string& exePath, const AZStd::string& commandLineArgs, const ArchiveResponseOutputCallback& respCallback, AZ::Uuid taskHandle = AZ::Uuid::CreateNull(), const AZStd::string& workingDir = "", bool captureOutput = false);
AZStd::string m_zipExePath;
AZStd::string m_unzipExePath;
private:
AZ::IO::FileIOBase* m_fileIO = nullptr;
AZ::IO::IArchive* m_archive = nullptr;
AZStd::vector<AZStd::thread> m_threads;
// Struct for tracking background threads/tasks
struct ThreadInfo
{
bool shouldStop = false;
AZStd::set<AZStd::thread::id> threads;
};
AZStd::mutex m_threadControlMutex; // Guards m_threadInfoMap
AZStd::condition_variable m_cv;
AZStd::unordered_map<AZ::Uuid, ThreadInfo> m_threadInfoMap;
bool CheckParamsForAdd(const AZStd::string& directory, const AZStd::string& file);
bool CheckParamsForExtract(const AZStd::string& archive, const AZStd::string& directory);
bool CheckParamsForCreate(const AZStd::string& archive, const AZStd::string& directory);
};
} // namespace AzToolsFramework
Code/Framework/AzToolsFramework/AzToolsFramework/Archive/NullArchiveComponent.cpp
+42 -71
View File
@@ -16,91 +16,62 @@ namespace AzToolsFramework
void NullArchiveComponent::Activate()
{
ArchiveCommands::Bus::Handler::BusConnect();
ArchiveCommandsBus::Handler::BusConnect();
}
void NullArchiveComponent::Deactivate()
{
ArchiveCommands::Bus::Handler::BusDisconnect();
ArchiveCommandsBus::Handler::BusDisconnect();
}
bool NullArchiveComponent::ExtractArchiveBlocking(const AZStd::string& /*archivePath*/, const AZStd::string& /*destinationPath*/, bool /*extractWithRootDirectory*/)
std::future<bool> DefaultFuture()
{
std::promise<bool> p;
p.set_value(false);
return p.get_future();
}
std::future<bool> NullArchiveComponent::CreateArchive(
const AZStd::string& /*archivePath*/,
const AZStd::string& /*dirToArchive*/)
{
return DefaultFuture();
}
std::future<bool> NullArchiveComponent::ExtractArchive(
const AZStd::string& /*archivePath*/,
const AZStd::string& /*destinationPath*/)
{
return DefaultFuture();
}
std::future<bool> NullArchiveComponent::ExtractFile(
const AZStd::string& /*archivePath*/,
const AZStd::string& /*fileInArchive*/,
const AZStd::string& /*destinationPath*/)
{
return DefaultFuture();
}
bool NullArchiveComponent::ListFilesInArchive(const AZStd::string& /*archivePath*/, AZStd::vector<AZStd::string>& /*outFileEntries*/)
{
return false;
}
void NullArchiveComponent::ExtractArchive(const AZStd::string& /*archivePath*/, const AZStd::string& /*destinationPath*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseCallback& respCallback)
std::future<bool> NullArchiveComponent::AddFileToArchive(
const AZStd::string& /*archivePath*/,
const AZStd::string& /*fileToAdd*/,
const AZStd::string& /*pathInArchive*/)
{
AZ::TickBus::QueueFunction(respCallback, false);
return DefaultFuture();
}
void NullArchiveComponent::ExtractArchiveOutput(const AZStd::string& /*archivePath*/, const AZStd::string& /*destinationPath*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
void NullArchiveComponent::ExtractArchiveWithoutRoot(const AZStd::string& /*archivePath*/, const AZStd::string& /*destinationPath*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
void NullArchiveComponent::ExtractFile(const AZStd::string& /*archivePath*/, const AZStd::string& /*fileInArchive*/, const AZStd::string& /*destinationPath*/, bool /*overWrite*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
// Always report we failed to extract
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
bool NullArchiveComponent::ExtractFileBlocking(const AZStd::string& /*archivePath*/, const AZStd::string& /*fileInArchive*/, const AZStd::string& /*destinationPath*/, bool /*overWrite*/)
{
return false;
}
void NullArchiveComponent::ListFilesInArchive(const AZStd::string& /*archivePath*/, AZStd::vector<AZStd::string>& /*consoleOutput*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
// Always report we failed to extract
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
bool NullArchiveComponent::ListFilesInArchiveBlocking(const AZStd::string& /*archivePath*/, AZStd::vector<AZStd::string>& /*consoleOutput*/)
{
return false;
}
void NullArchiveComponent::AddFileToArchive(const AZStd::string& /*archivePath*/, const AZStd::string& /*fileToAdd*/, const AZStd::string& /*pathInArchive*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
// Always report we failed to extract
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
bool NullArchiveComponent::AddFileToArchiveBlocking(const AZStd::string& /*archivePath*/, const AZStd::string& /*fileToAdd*/, const AZStd::string& /*pathInArchive*/)
{
return false;
}
bool NullArchiveComponent::AddFilesToArchiveBlocking(const AZStd::string& /*archivePath*/, const AZStd::string& /*workingDirectory*/, const AZStd::string& /*listFilePath*/)
{
return false;
}
void NullArchiveComponent::AddFilesToArchive(const AZStd::string& /*archivePath*/, const AZStd::string& /*workingDirectory*/, const AZStd::string& /*listFilePath*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
// Always report we failed to extract
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
void NullArchiveComponent::CreateArchive(const AZStd::string& /*archivePath*/, const AZStd::string& /*dirToArchive*/, AZ::Uuid /*taskHandle*/, const ArchiveResponseOutputCallback& respCallback)
{
// Always report we failed to extract
AZ::TickBus::QueueFunction(respCallback, false, AZStd::string());
}
bool NullArchiveComponent::CreateArchiveBlocking(const AZStd::string& /*archivePath*/, const AZStd::string& /*dirToArchive*/)
{
return false;
}
void NullArchiveComponent::CancelTasks(AZ::Uuid /*taskHandle*/)
std::future<bool> NullArchiveComponent::AddFilesToArchive(
const AZStd::string& /*archivePath*/,
const AZStd::string& /*workingDirectory*/,
const AZStd::string& /*listFilePath*/)
{
return DefaultFuture();
}
void NullArchiveComponent::Reflect(AZ::ReflectContext* context)
Code/Framework/AzToolsFramework/AzToolsFramework/Archive/NullArchiveComponent.h
+26 -18
View File
@@ -15,7 +15,7 @@ namespace AzToolsFramework
{
class NullArchiveComponent
: public AZ::Component
, private ArchiveCommands::Bus::Handler
, private ArchiveCommandsBus::Handler
{
public:
AZ_COMPONENT(NullArchiveComponent, "{D665B6B1-5FF4-4203-B19F-BBDB82587129}")
@@ -32,23 +32,31 @@ namespace AzToolsFramework
static void Reflect(AZ::ReflectContext* context);
//////////////////////////////////////////////////////////////////////////
// ArchiveCommands::Bus::Handler overrides
// ArchiveCommands::Bus::Handler overrides
void CreateArchive(const AZStd::string& archivePath, const AZStd::string& dirToArchive, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool CreateArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& dirToArchive) override;
bool ExtractArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool extractWithRootDirectory) override;
void ExtractArchive(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseCallback& respCallback) override;
void ExtractArchiveOutput(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
void ExtractArchiveWithoutRoot(const AZStd::string& archivePath, const AZStd::string& destinationPath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
void ExtractFile(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool ExtractFileBlocking(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite) override;
void ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& consoleOutput, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool ListFilesInArchiveBlocking(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& consoleOutput) override;
void AddFileToArchive(const AZStd::string& archivePath, const AZStd::string& fileToAdd, const AZStd::string& pathInArchive, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
bool AddFileToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& fileToAdd, const AZStd::string& pathInArchive) override;
bool AddFilesToArchiveBlocking(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath) override;
void AddFilesToArchive(const AZStd::string& archivePath, const AZStd::string& workingDirectory, const AZStd::string& listFilePath, AZ::Uuid taskHandle, const ArchiveResponseOutputCallback& respCallback) override;
void CancelTasks(AZ::Uuid taskHandle) override;
// ArchiveCommandsBus::Handler overrides
[[nodiscard]] std::future<bool> CreateArchive(
const AZStd::string& archivePath,
const AZStd::string& dirToArchive) override;
[[nodiscard]] std::future<bool> ExtractArchive(
const AZStd::string& archivePath,
const AZStd::string& destinationPath) override;
[[nodiscard]] std::future<bool> ExtractFile(
const AZStd::string& archivePath,
const AZStd::string& fileInArchive,
const AZStd::string& destinationPath) override;
bool ListFilesInArchive(const AZStd::string& archivePath, AZStd::vector<AZStd::string>& outFileEntries) override;
[[nodiscard]] std::future<bool> AddFileToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& fileToAdd) override;
[[nodiscard]] std::future<bool> AddFilesToArchive(
const AZStd::string& archivePath,
const AZStd::string& workingDirectory,
const AZStd::string& listFilePath) override;
//////////////////////////////////////////////////////////////////////////
};
} // namespace AzToolsFramework
Code/Framework/AzToolsFramework/AzToolsFramework/AssetBundle/AssetBundleComponent.cpp
+17 -10
View File
@@ -32,12 +32,10 @@ namespace AzToolsFramework
const int NumOfBytesInMB = 1024 * 1024;
const int ManifestFileSizeBufferInBytes = 10 * 1024; // 10 KB
const float AssetCatalogFileSizeBufferPercentage = 1.0f;
using ArchiveCommandsBus = AzToolsFramework::ArchiveCommands::Bus;
using AssetCatalogRequestBus = AZ::Data::AssetCatalogRequestBus;
const char AssetBundleComponent::DeltaCatalogName[] = "DeltaCatalog.xml";
constexpr int SleepTimeMS = 250;
constexpr int InjectFileRetryCount = 4;
@@ -136,7 +134,7 @@ namespace AzToolsFramework
AZ_TracePrintf(logWindowName, "Gathering file entries in source pak file \"%s\".\n", sourcePak.c_str());
bool result = false;
AZStd::vector<AZStd::string> fileEntries;
ArchiveCommandsBus::BroadcastResult(result, &AzToolsFramework::ArchiveCommands::ListFilesInArchiveBlocking, normalizedSourcePakPath, fileEntries);
ArchiveCommandsBus::BroadcastResult(result, &AzToolsFramework::ArchiveCommandsBus::Events::ListFilesInArchive, normalizedSourcePakPath, fileEntries);
// This ebus currently always returns false as the result, as it is believed that the 7z process is
// being terminated by the user instead of ending gracefully. Check against an empty fileList instead
// as a result.
@@ -606,15 +604,17 @@ namespace AzToolsFramework
{
AZ_TracePrintf(logWindowName, "Injecting file (%s) into bundle (%s).\n", filePath.c_str(), archiveFilePath.c_str());
bool fileAddedToArchive = false;
std::future<bool> fileAdded;
int retryCount = InjectFileRetryCount;
while (!fileAddedToArchive && retryCount)
{
ArchiveCommandsBus::BroadcastResult(fileAddedToArchive, &AzToolsFramework::ArchiveCommands::AddFileToArchiveBlocking, archiveFilePath, workingDirectory, filePath);
ArchiveCommandsBus::BroadcastResult(fileAdded, &AzToolsFramework::ArchiveCommandsBus::Events::AddFileToArchive, archiveFilePath, workingDirectory, filePath);
--retryCount;
fileAddedToArchive = fileAdded.get();
if (!fileAddedToArchive && retryCount)
{
AZ_Error(logWindowName, false, "Failed to insert file (%s) into bundle (%s). Retrying.", filePath.c_str(), archiveFilePath.c_str());
AZStd::this_thread::sleep_for(AZStd::chrono::milliseconds(SleepTimeMS));
}
}
@@ -627,7 +627,11 @@ namespace AzToolsFramework
bool AssetBundleComponent::InjectFile(const AZStd::string& filePath, const AZStd::string& sourcePak)
{
return InjectFile(filePath, sourcePak, "");
// When no working directory is specified, assume that the file being injected goes into the root of the archive.
// The filePath should be an absolute path, making the workingDirectory be the path leading up to the file.
AZ::IO::PathView fullFilePath{ filePath, AZ::IO::PosixPathSeparator };
AZ::IO::Path workingDir{ fullFilePath.ParentPath() };
return InjectFile(filePath, sourcePak, workingDir.c_str());
}
bool AssetBundleComponent::InjectFiles(const AZStd::vector<AZStd::string>& fileEntries, const AZStd::string& sourcePak, const char* workingDirectory)
@@ -668,8 +672,9 @@ namespace AzToolsFramework
}
}
bool filesAddedToArchive = false;
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(filesAddedToArchive, &AzToolsFramework::ArchiveCommands::AddFilesToArchiveBlocking, sourcePak, workingDirectory, listFilePath);
std::future<bool> filesAdded;
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(filesAdded, &AzToolsFramework::ArchiveCommands::AddFilesToArchive, sourcePak, workingDirectory, listFilePath);
bool filesAddedToArchive = filesAdded.get();
if (!filesAddedToArchive)
{
AZ_Error(logWindowName, false, "Failed to insert files into bundle (%s).\n", sourcePak.c_str());
@@ -688,7 +693,6 @@ namespace AzToolsFramework
{
// open the manifest and deserialize it
bool manifestExtracted = false;
const bool overwriteExisting = true;
TemporaryDir tempDir(sourcePak);
if (!tempDir.m_result)
@@ -698,7 +702,10 @@ namespace AzToolsFramework
AZStd::string manifestFilePath;
AzFramework::StringFunc::Path::ConstructFull(tempDir.m_tempFolderPath.c_str(), AzFramework::AssetBundleManifest::s_manifestFileName, manifestFilePath, true);
ArchiveCommandsBus::BroadcastResult(manifestExtracted, &ArchiveCommandsBus::Events::ExtractFileBlocking, sourcePak, AzFramework::AssetBundleManifest::s_manifestFileName, tempDir.m_tempFolderPath, overwriteExisting);
std::future<bool> extractResult;
ArchiveCommandsBus::BroadcastResult(extractResult, &ArchiveCommandsBus::Events::ExtractFile, sourcePak, AzFramework::AssetBundleManifest::s_manifestFileName, tempDir.m_tempFolderPath);
manifestExtracted = extractResult.get();
if (!manifestExtracted)
{
AZ_Error(logWindowName, false, "Failed to extract existing manifest from archive \"%s\".", sourcePak.c_str());
Code/Framework/AzToolsFramework/AzToolsFramework/AssetBundle/AssetBundleComponent.h
+1 -1
View File
@@ -57,7 +57,7 @@ namespace AzToolsFramework
//! Returns true if the file at filePath was successfully injected into the bundle at sourcePak
static bool InjectFile(const AZStd::string& filePath, const AZStd::string& sourcePak, const char* workingDirectory);
//! Inject the files with relative filePaths which espect to the working directory into the bundle at sourcePak
//! Inject the files with relative filePaths with respect to the working directory into the bundle at sourcePak
//! Returns true if the file at filePath was successfully injected into the bundle at sourcePak
static bool InjectFiles(const AZStd::vector<AZStd::string>& fileEntries, const AZStd::string& sourcePak, const char* workingDirectory);
Code/Framework/AzToolsFramework/Platform/Linux/AzToolsFramework/Archive/ArchiveComponent_Linux.cpp
-230
View File
@@ -1,230 +0,0 @@
/*
* Copyright (c) Contributors to the Open 3D Engine Project.
* For complete copyright and license terms please see the LICENSE at the root of this distribution.
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*
*/
#include <AzCore/std/string/string.h>
#include <AzCore/Outcome/Outcome.h>
#include <AzFramework/StringFunc/StringFunc.h>
#include <AzFramework/IO/LocalFileIO.h>
namespace AzToolsFramework
{
namespace Platform
{
[[maybe_unused]] static const char ErrorChannel[] = "ArchiveComponent_Linux";
static const char ZipExePath[] = R"(/usr/bin/zip)";
static const char UnzipExePath[] = R"(/usr/bin/unzip)";
static const char CreateArchiveCmd[] = "-r \"%s\" . -i *";
static const char ExtractArchiveCmd[] = R"(-o "%s" -d "%s")";
static const char AddFileCmd[] = R"("%s" "%s")";
static const char ExtractFileCmd[] = R"(%s "%s" %s)";
static const char ExtractFileDestination[] = R"(%s "%s" "%s" -d "%s")";
static const char ExtractOverwrite[] = "-o";
static const char ExtractSkipExisting[] = "-n";
static const char ListFilesInArchiveCmd[] = "-l %s";
AZStd::string GetZipExePath()
{
return ZipExePath;
}
AZStd::string GetUnzipExePath()
{
return UnzipExePath;
}
AZ::Outcome<AZStd::string, AZStd::string> MakePath(const AZStd::string& path)
{
// Create the folder if it does not already exist
if (!AZ::IO::FileIOBase::GetInstance()->Exists(path.c_str()))
{
auto result = AZ::IO::FileIOBase::GetInstance()->CreatePath(path.c_str());
if (!result)
{
return AZ::Failure(AZStd::string::format("Path creation failed. Input path: %s \n", path.c_str()));
}
}
return AZ::Success(path);
}
AZ::Outcome<AZStd::string, AZStd::string> MakeCreateArchivePath(const AZStd::string& archivePath)
{
// Remove the file name from the input path
// /some/folder/path/archive.zip -> /some/folder/path/
AZStd::string strippedArchivePath = archivePath;
AzFramework::StringFunc::Path::StripFullName(strippedArchivePath);
if (strippedArchivePath.empty())
{
return AZ::Failure(AZStd::string::format("Stripped path name is empty. Cancelling path creation. Input path: %s\n", archivePath.c_str()));
}
return MakePath(strippedArchivePath);
}
AZ::Outcome<AZStd::string, AZStd::string> MakeExtractArchivePath(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
{
if(!includeRoot)
{
// Create the folder for the input destination path with no modifications
// /path/to/destination/
return MakePath(destinationPath);
}
// Get the name of the input archive. This will be the name of the root folder for the archive extraction
// /some/folder/path/archive.zip -> archive
AZStd::string zipFileName;
bool result = AzFramework::StringFunc::Path::GetFileName(archivePath.c_str(), zipFileName);
if(!result)
{
return AZ::Failure(AZStd::string::format("Failed to get name of zip file from the archive path. Cancelling path creation. \n Input Archive Path: %s \n", archivePath.c_str()));
}
// Append the root folder name to the end of the destination path
// /path/to/destination/ + archive -> /path/to/destination/archive
AZStd::string destinationPathWithRoot;
result = AzFramework::StringFunc::Path::Join(destinationPath.c_str(), zipFileName.c_str(), destinationPathWithRoot);
if(!result)
{
return AZ::Failure(AZStd::string::format("Failed to append zip file name to the destination path. Cancelling path creation. \n Destination Path: %s \n Zip file name: %s \n", destinationPath.c_str(), zipFileName.c_str()));
}
// Append a separator so that it is formatted like a folder
// /path/to/destination/archive -> /path/to/destination/archive/
AzFramework::StringFunc::Path::AppendSeparator(destinationPathWithRoot);
return MakePath(destinationPathWithRoot);
}
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
{
auto pathCreationResult = MakeCreateArchivePath(archivePath);
if (!pathCreationResult)
{
AZ_Error(ErrorChannel, false, "%s", pathCreationResult.GetError().c_str());
return "";
}
AZ_UNUSED(dirToArchive);
return AZStd::string::format(CreateArchiveCmd, archivePath.c_str());
}
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
{
auto pathCreationResult = MakeExtractArchivePath(archivePath, destinationPath, includeRoot);
if (!pathCreationResult)
{
AZ_Error(ErrorChannel, false, "%s", pathCreationResult.GetError().c_str());
return "";
}
return AZStd::string::format(ExtractArchiveCmd, archivePath.c_str(), pathCreationResult.GetValue().c_str());
}
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& /*archivePath*/, const AZStd::string& /*listFilePath*/)
{
// Adding files into a archive using a list file is not currently supported
return {};
}
bool IsAddFilesToArchiveCommandSupported()
{
// Adding files into a archive using a list file is not currently supported
return false;
}
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file)
{
if (!MakeCreateArchivePath(archivePath).IsSuccess())
{
AZ_Error(ErrorChannel, false, "Unable to make path for ( %s ).\n", archivePath.c_str());
return {};
}
return AZStd::string::format(AddFileCmd, archivePath.c_str(), file.c_str());
}
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
{
AZStd::string commandLineArgs;
if (destinationPath.empty())
{
// Extract file in archive from archive path to the current directory, overwriting a file of the same name that exists there.
commandLineArgs = AZStd::string::format(ExtractFileCmd, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str());
}
else
{
if (!MakePath(destinationPath).IsSuccess())
{
AZ_Error(ErrorChannel, false, "Unable to make path ( %s ).\n", destinationPath.c_str());
return {};
}
// Extract file in archive from archive path to destinationPath, overwriting a file of the same name that exists there.
commandLineArgs = AZStd::string::format(ExtractFileDestination, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str(), destinationPath.c_str());
}
return commandLineArgs;
}
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath)
{
AZStd::string commandLineArgs = AZStd::string::format(ListFilesInArchiveCmd, archivePath.c_str());
return commandLineArgs;
}
/*
Sample Console Output of the unzip list command
Archive: /var/folders/1q/12nyzqc913qgm532y2c98mnm6w4_qv/T/ArchiveTests-ra8oMy/TestArchive.pak
Length Date Time Name
--------- ---------- ----- ----
0 10-14-2019 15:22 testfolder/
1 10-14-2019 15:22 testfolder/folderfile.txt
1 10-14-2019 15:22 basicfile.txt
1 10-14-2019 15:22 basicfile2.txt
0 10-14-2019 15:22 testfolder2/
1 10-14-2019 15:22 testfolder2/sharedfolderfile2.txt
1 10-14-2019 15:22 testfolder2/sharedfolderfile.txt
0 10-14-2019 15:22 testfolder3/
0 10-14-2019 15:22 testfolder3/testfolder4/
1 10-14-2019 15:22 testfolder3/testfolder4/depthfile.bat
--------- -------
6 10 files
*/
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries)
{
AZStd::vector<AZStd::string> fileEntryData;
AzFramework::StringFunc::Tokenize(consoleOutput.c_str(), fileEntryData, "\n");
int startingLineIdx = 3; // first line that might contain the file name
for (size_t lineIdx = startingLineIdx; lineIdx < fileEntryData.size(); ++lineIdx)
{
AZStd::string& line = fileEntryData[lineIdx];
AZStd::vector<AZStd::string> lineEntryData;
AzFramework::StringFunc::Tokenize(line.c_str(), lineEntryData, " ");
AZStd::string& fileName = lineEntryData.back();
if(fileName.back() == AZ_CORRECT_FILESYSTEM_SEPARATOR)
{
// if the filename ends with a separator
// than it indicates that this is a directory
continue;
}
if(fileName.compare("-------") == 0)
{
return;
}
fileEntries.emplace_back(fileName);
}
}
} // namespace Platform
} // namespace AzToolsFramework
Code/Framework/AzToolsFramework/Platform/Linux/platform_linux_files.cmake
-1
View File
@@ -7,5 +7,4 @@
#
set(FILES
AzToolsFramework/Archive/ArchiveComponent_Linux.cpp
)
Code/Framework/AzToolsFramework/Platform/Mac/AzToolsFramework/Archive/ArchiveComponent_Mac.cpp
-263
View File
@@ -1,263 +0,0 @@
/*
* Copyright (c) Contributors to the Open 3D Engine Project.
* For complete copyright and license terms please see the LICENSE at the root of this distribution.
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*
*/
#include <AzCore/std/string/string.h>
#include <AzCore/Outcome/Outcome.h>
#include <AzFramework/StringFunc/StringFunc.h>
#include <AzFramework/IO/LocalFileIO.h>
namespace AzToolsFramework
{
namespace Platform
{
const char ErrorChannel[] = "ArchiveComponent_OSX";
const char ZipExePath[] = R"(/usr/bin/zip)";
const char UnzipExePath[] = R"(/usr/bin/unzip)";
// v Requires investigation, the correct cmd should be R"(-r "%s" "%s/")" but tests fail
const char CreateArchiveCmd[] = R"(-r "%s" .)";
const char ExtractArchiveCmd[] = R"(-o "%s" -d "%s")";
const char AddFileCmd[] = R"("%s" "%s" -X)";
const char AddFilesCmd[] = R"("%s" -X %s)";
const char ExtractFileCmd[] = R"(%s "%s" %s)";
const char ExtractFileDestination[] = R"(%s "%s" "%s" -d "%s")";
const char ExtractOverwrite[] = "-o";
const char ExtractSkipExisting[] = "-n";
const char ListFilesInArchiveCmd[] = "-l %s";
AZStd::string GetZipExePath()
{
return ZipExePath;
}
AZStd::string GetUnzipExePath()
{
return UnzipExePath;
}
AZ::Outcome<AZStd::string, AZStd::string> MakePath(const AZStd::string& path)
{
// Create the folder if it does not already exist
if (!AZ::IO::FileIOBase::GetInstance()->Exists(path.c_str()))
{
auto result = AZ::IO::FileIOBase::GetInstance()->CreatePath(path.c_str());
if (!result)
{
return AZ::Failure(AZStd::string::format("Path creation failed. Input path: %s \n", path.c_str()));
}
}
return AZ::Success(path);
}
AZ::Outcome<AZStd::string, AZStd::string> MakeCreateArchivePath(const AZStd::string& archivePath)
{
// Remove the file name from the input path
// /some/folder/path/archive.zip -> /some/folder/path/
AZStd::string strippedArchivePath = archivePath;
AzFramework::StringFunc::Path::StripFullName(strippedArchivePath);
if (strippedArchivePath.empty())
{
return AZ::Failure(AZStd::string::format("Stripped path name is empty. Cancelling path creation. Input path: %s\n", archivePath.c_str()));
}
return MakePath(strippedArchivePath);
}
AZ::Outcome<AZStd::string, AZStd::string> MakeExtractArchivePath(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
{
if(!includeRoot)
{
// Create the folder for the input destination path with no modifications
// /path/to/destination/
return MakePath(destinationPath);
}
// Get the name of the input archive. This will be the name of the root folder for the archive extraction
// /some/folder/path/archive.zip -> archive
AZStd::string zipFileName;
bool result = AzFramework::StringFunc::Path::GetFileName(archivePath.c_str(), zipFileName);
if(!result)
{
return AZ::Failure(AZStd::string::format("Failed to get name of zip file from the archive path. Cancelling path creation. \n Input Archive Path: %s \n", archivePath.c_str()));
}
// Append the root folder name to the end of the destination path
// /path/to/destination/ + archive -> /path/to/destination/archive
AZStd::string destinationPathWithRoot;
result = AzFramework::StringFunc::Path::Join(destinationPath.c_str(), zipFileName.c_str(), destinationPathWithRoot);
if(!result)
{
return AZ::Failure(AZStd::string::format("Failed to append zip file name to the destination path. Cancelling path creation. \n Destination Path: %s \n Zip file name: %s \n", destinationPath.c_str(), zipFileName.c_str()));
}
// Append a separator so that it is formatted like a folder
// /path/to/destination/archive -> /path/to/destination/archive/
AzFramework::StringFunc::Path::AppendSeparator(destinationPathWithRoot);
return MakePath(destinationPathWithRoot);
}
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
{
auto pathCreationResult = MakeCreateArchivePath(archivePath);
if (!pathCreationResult.IsSuccess())
{
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
return "";
}
// LY-116692. Requires proper investigation, the correct format should be:
// AZStd::string::format(CreateArchiveCmd, archivePath.c_str(), dirToArchive.c_str());
// but unit test ArchiveTest.ListFilesInArchiveBlocking_FilesAtThreeDepths_FilesFound fails
AZ_UNUSED(dirToArchive);
return AZStd::string::format(CreateArchiveCmd, archivePath.c_str());
}
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
{
auto pathCreationResult = MakeExtractArchivePath(archivePath, destinationPath, includeRoot);
if (!pathCreationResult)
{
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
return "";
}
return AZStd::string::format(ExtractArchiveCmd, archivePath.c_str(), pathCreationResult.GetValue().c_str());
}
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& listFilePath)
{
auto pathCreationResult = MakeCreateArchivePath(archivePath);
if (!pathCreationResult)
{
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
return "";
}
AZStd::string fileListStr;
{
AZ::IO::FileIOStream fileStream(listFilePath.c_str(), AZ::IO::OpenMode::ModeRead | AZ::IO::OpenMode::ModeText);
if (fileStream.IsOpen())
{
AZ::IO::SizeType length = fileStream.GetLength();
AZStd::vector<char> charBuffer;
charBuffer.resize_no_construct(length + 1);
fileStream.Read(length, charBuffer.data());
charBuffer.back() = 0;
fileListStr.append("\"");
fileListStr.insert(1, charBuffer.data());
AzFramework::StringFunc::Replace(fileListStr, "\n", "\" \"");
fileListStr.append("\"");
}
else
{
AZ_Error(ErrorChannel, false, "Unable to read list file ( %s ) \n", listFilePath.c_str());
return "";
}
}
return AZStd::string::format(AddFilesCmd, archivePath.c_str(), fileListStr.c_str());
}
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file)
{
auto pathCreationResult = MakeCreateArchivePath(archivePath);
if (!pathCreationResult)
{
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
return "";
}
return AZStd::string::format(AddFileCmd, archivePath.c_str(), file.c_str());
}
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
{
AZStd::string commandLineArgs;
if (destinationPath.empty())
{
// Extract file in archive from archive path to the current directory, overwriting a file of the same name that exists there.
commandLineArgs = AZStd::string::format(ExtractFileCmd, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str());
}
else
{
if (!MakePath(destinationPath).IsSuccess())
{
AZ_Error(ErrorChannel, false, "Unable to make path ( %s ).\n", destinationPath.c_str());
return "";
}
// Extract file in archive from archive path to destinationPath, overwriting a file of the same name that exists there.
commandLineArgs = AZStd::string::format(ExtractFileDestination, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str(), destinationPath.c_str());
}
return commandLineArgs;
}
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath)
{
AZStd::string commandLineArgs = AZStd::string::format(ListFilesInArchiveCmd, archivePath.c_str());
return commandLineArgs;
}
/*
Sample Console Output of the unzip list command
Archive: /var/folders/1q/12nyzqc913qgm532y2c98mnm6w4_qv/T/ArchiveTests-ra8oMy/TestArchive.pak
Length Date Time Name
--------- ---------- ----- ----
0 10-14-2019 15:22 testfolder/
1 10-14-2019 15:22 testfolder/folderfile.txt
1 10-14-2019 15:22 basicfile.txt
1 10-14-2019 15:22 basicfile2.txt
0 10-14-2019 15:22 testfolder2/
1 10-14-2019 15:22 testfolder2/sharedfolderfile2.txt
1 10-14-2019 15:22 testfolder2/sharedfolderfile.txt
0 10-14-2019 15:22 testfolder3/
0 10-14-2019 15:22 testfolder3/testfolder4/
1 10-14-2019 15:22 testfolder3/testfolder4/depthfile.bat
--------- -------
6 10 files
*/
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries)
{
AZStd::vector<AZStd::string> fileEntryData;
AzFramework::StringFunc::Tokenize(consoleOutput.c_str(), fileEntryData, "\n");
int startingLineIdx = 3; // first line that might contain the file name
for (size_t lineIdx = startingLineIdx; lineIdx < fileEntryData.size(); ++lineIdx)
{
AZStd::string& line = fileEntryData[lineIdx];
AZStd::vector<AZStd::string> lineEntryData;
AzFramework::StringFunc::Tokenize(line.c_str(), lineEntryData, " ");
AZStd::string& fileName = lineEntryData.back();
if(fileName.back() == AZ_CORRECT_FILESYSTEM_SEPARATOR)
{
// if the filename ends with a separator
// than it indicates that this is a directory
continue;
}
if(fileName.compare("-------") == 0)
{
return;
}
fileEntries.emplace_back(fileName);
}
}
} // namespace Platform
} // namespace AzToolsFramework
Code/Framework/AzToolsFramework/Platform/Mac/platform_mac_files.cmake
-1
View File
@@ -7,5 +7,4 @@
#
set(FILES
AzToolsFramework/Archive/ArchiveComponent_Mac.cpp
)
Code/Framework/AzToolsFramework/Platform/Windows/AzToolsFramework/Archive/ArchiveComponent_Windows.cpp
-167
View File
@@ -1,167 +0,0 @@
/*
* Copyright (c) Contributors to the Open 3D Engine Project.
* For complete copyright and license terms please see the LICENSE at the root of this distribution.
*
* SPDX-License-Identifier: Apache-2.0 OR MIT
*
*/
#include <AzCore/std/string/string.h>
#include <AzCore/Component/ComponentApplicationBus.h>
#include <AzFramework/StringFunc/StringFunc.h>
#include <AzToolsFramework/Archive/ArchiveComponent.h>
namespace AzToolsFramework
{
namespace Platform
{
const char CreateArchiveCmd[] = R"(a -tzip -mx=1 "%s" -r "%s\*")";
// -aos is for skipping extract on existing files
const char ExtractArchiveCmd[] = R"(x -mmt=off "%s" -o"%s\*" -aos)";
const char ExtractArchiveWithoutRootCmd[] = R"(x -mmt=off "%s" -o"%s" -aos)";
const char AddFilesCmd[] = R"(a -tzip "%s" @"%s")";
const char AddFileCmd[] = R"(a -tzip "%s" "%s")";
const char ExtractFileCmd[] = R"(e -mmt=off "%s" "%s" %s)";
const char ExtractFileDestination[] = R"(e -mmt=off "%s" -o"%s" "%s" %s)";
const char ExtractOverwrite[] = "-aoa";
const char ExtractSkipExisting[] = "-aos";
const char ListFilesInArchiveCmd[] = R"(l -r -slt "%s")";
AZStd::string Get7zExePath()
{
const char* rootPath = nullptr;
AZ::ComponentApplicationBus::BroadcastResult(rootPath, &AZ::ComponentApplicationRequests::GetEngineRoot);
AZStd::string exePath;
AzFramework::StringFunc::Path::ConstructFull(rootPath, "Tools", "7za", ".exe", exePath);
return exePath;
}
AZStd::string GetZipExePath()
{
return Get7zExePath();
}
AZStd::string GetUnzipExePath()
{
return Get7zExePath();
}
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
{
return AZStd::string::format(CreateArchiveCmd, archivePath.c_str(), dirToArchive.c_str());
}
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
{
if (includeRoot)
{
// Extract archive path to destinationPath\<archiveFileName> and skipping extracting of existing files
return AZStd::string::format(ExtractArchiveCmd, archivePath.c_str(), destinationPath.c_str());
}
else
{
// Extract archive path to destinationPath and skipping extracting of existing files
return AZStd::string::format(ExtractArchiveWithoutRootCmd, archivePath.c_str(), destinationPath.c_str());
}
}
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& listFilePath)
{
return AZStd::string::format(AddFilesCmd, archivePath.c_str(), listFilePath.c_str());
}
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file)
{
return AZStd::string::format(AddFileCmd, archivePath.c_str(), file.c_str());
}
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
{
AZStd::string commandLineArgs;
if (destinationPath.empty())
{
// Extract file in archive from archive path to the current directory, overwriting a file of the same name that exists there.
commandLineArgs = AZStd::string::format(ExtractFileCmd, archivePath.c_str(), fileInArchive.c_str(), overWrite ? ExtractOverwrite : ExtractSkipExisting);
}
else
{
// Extract file in archive from archive path to destinationPath, overwriting a file of the same name that exists there.
commandLineArgs = AZStd::string::format(ExtractFileDestination, archivePath.c_str(), destinationPath.c_str(), fileInArchive.c_str(), overWrite ? ExtractOverwrite : ExtractSkipExisting);
}
return commandLineArgs;
}
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath)
{
AZStd::string commandLineArgs = AZStd::string::format(ListFilesInArchiveCmd, archivePath.c_str());
return commandLineArgs;
}
/*
File output for our list archive commands takes the following two patterns for files vs directories:
Path = basicfile2.txt
Folder = -
Size = 1
Packed Size = 1
Modified = 2019-03-26 18:31:10
Created = 2019-03-26 18:31:10
Accessed = 2019-03-26 18:31:10
Attributes = A
Encrypted = -
Comment =
CRC = 32D70693
Method = Store
Characteristics = NTFS
Host OS = FAT
Version = 10
Volume Index = 0
Offset = 44
Path = testfolder
Folder = +
Size = 0
Packed Size = 0
Modified = 2019-03-26 18:31:10
Created = 2019-03-26 18:31:10
Accessed = 2019-03-26 18:31:10
Attributes = D
Encrypted = -
Comment =
CRC =
Method = Store
Characteristics = NTFS
Host OS = FAT
Version = 20
Volume Index = 0
Offset = 89
*/
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries)
{
AZStd::vector<AZStd::string> fileEntryData;
AzFramework::StringFunc::Tokenize(consoleOutput.c_str(), fileEntryData, "\r\n");
for (size_t slotNum = 0; slotNum < fileEntryData.size(); ++slotNum)
{
AZStd::string& line = fileEntryData[slotNum];
if (AzFramework::StringFunc::StartsWith(line, "Path = "))
{
if ((slotNum + 1) < fileEntryData.size())
{
// We're checking one past each entry we find for the Folder entry and skipping anything marked as a folder
// See sample output above
if (AzFramework::StringFunc::StartsWith(fileEntryData[slotNum + 1], "Folder = -"))
{
AzFramework::StringFunc::Replace(line, "Path = ", "", false, true);
fileEntries.emplace_back(AZStd::move(line));
slotNum++;
}
}
}
}
}
} // namespace Platform
} // namespace AzToolsFramework
Code/Framework/AzToolsFramework/Platform/Windows/platform_windows_files.cmake
-1
View File
@@ -7,5 +7,4 @@
#
set(FILES
AzToolsFramework/Archive/ArchiveComponent_Windows.cpp
)
Code/Framework/AzToolsFramework/Tests/ArchiveTests.cpp
+104 -22
View File
@@ -31,7 +31,6 @@ namespace UnitTest
{
namespace
{
bool CreateDummyFile(const QString& fullPathToFile, const QString& tempStr = {})
{
QFileInfo fi(fullPathToFile);
@@ -50,7 +49,7 @@ namespace UnitTest
return true;
}
class ArchiveTest :
class ArchiveComponentTest :
public ::testing::Test
{
@@ -73,7 +72,12 @@ namespace UnitTest
return "Archive";
}
void CreateArchiveFolder( QString archiveFolderName, QStringList fileList )
QString GetExtractFolderName()
{
return "Extracted";
}
void CreateArchiveFolder(QString archiveFolderName, QStringList fileList)
{
QDir tempPath = QDir(m_tempDir.GetDirectory()).filePath(archiveFolderName);
@@ -84,6 +88,14 @@ namespace UnitTest
}
}
QString CreateArchiveListTextFile()
{
QString listFilePath = QDir(m_tempDir.GetDirectory()).absoluteFilePath("FileList.txt");
QString textContent = CreateArchiveFileList().join("\n");
EXPECT_TRUE(CreateDummyFile(listFilePath, textContent));
return listFilePath;
}
void CreateArchiveFolder()
{
CreateArchiveFolder(GetArchiveFolderName(), CreateArchiveFileList());
@@ -99,16 +111,24 @@ namespace UnitTest
return QDir(m_tempDir.GetDirectory()).filePath(GetArchiveFolderName());
}
QString GetExtractFolder()
{
return QDir(m_tempDir.GetDirectory()).filePath(GetExtractFolderName());
}
bool CreateArchive()
{
bool createResult{ false };
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(createResult, &AzToolsFramework::ArchiveCommandsBus::Events::CreateArchiveBlocking, GetArchivePath().toStdString().c_str(), GetArchiveFolder().toStdString().c_str());
return createResult;
std::future<bool> createResult;
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(createResult,
&AzToolsFramework::ArchiveCommandsBus::Events::CreateArchive,
GetArchivePath().toUtf8().constData(), GetArchiveFolder().toUtf8().constData());
bool result = createResult.get();
return result;
}
void SetUp() override
{
m_app.reset(aznew ToolsTestApplication("ArchiveTest"));
m_app.reset(aznew ToolsTestApplication("ArchiveComponentTest"));
m_app->Start(AzFramework::Application::Descriptor());
// Without this, the user settings component would attempt to save on finalize/shutdown. Since the file is
// shared across the whole engine, if multiple tests are run in parallel, the saving could cause a crash
@@ -132,76 +152,138 @@ namespace UnitTest
};
#if AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
TEST_F(ArchiveTest, DISABLED_CreateArchiveBlocking_FilesAtThreeDepths_ArchiveCreated)
TEST_F(ArchiveComponentTest, DISABLED_CreateArchive_FilesAtThreeDepths_ArchiveCreated)
#else
TEST_F(ArchiveTest, CreateArchiveBlocking_FilesAtThreeDepths_ArchiveCreated)
TEST_F(ArchiveComponentTest, CreateArchive_FilesAtThreeDepths_ArchiveCreated)
#endif // AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
{
EXPECT_TRUE(m_tempDir.IsValid());
CreateArchiveFolder();
AZ_TEST_START_TRACE_SUPPRESSION;
bool createResult = CreateArchive();
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_EQ(createResult, true);
EXPECT_TRUE(createResult);
}
#if AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
TEST_F(ArchiveTest, DISABLED_ListFilesInArchiveBlocking_FilesAtThreeDepths_FilesFound)
TEST_F(ArchiveComponentTest, DISABLED_ListFilesInArchive_FilesAtThreeDepths_FilesFound)
#else
TEST_F(ArchiveTest, ListFilesInArchiveBlocking_FilesAtThreeDepths_FilesFound)
TEST_F(ArchiveComponentTest, ListFilesInArchive_FilesAtThreeDepths_FilesFound)
#endif // AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
{
EXPECT_TRUE(m_tempDir.IsValid());
CreateArchiveFolder();
AZ_TEST_START_TRACE_SUPPRESSION;
EXPECT_EQ(CreateArchive(), true);
AZStd::vector<AZStd::string> fileList;
bool listResult{ false };
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(listResult, &AzToolsFramework::ArchiveCommandsBus::Events::ListFilesInArchiveBlocking, GetArchivePath().toStdString().c_str(), fileList);
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(listResult,
&AzToolsFramework::ArchiveCommandsBus::Events::ListFilesInArchive,
GetArchivePath().toUtf8().constData(), fileList);
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_TRUE(listResult);
EXPECT_EQ(fileList.size(), 6);
}
#if AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
TEST_F(ArchiveTest, DISABLED_CreateDeltaCatalog_AssetsNotRegistered_Failure)
TEST_F(ArchiveComponentTest, DISABLED_CreateDeltaCatalog_AssetsNotRegistered_Failure)
#else
TEST_F(ArchiveTest, CreateDeltaCatalog_AssetsNotRegistered_Failure)
TEST_F(ArchiveComponentTest, CreateDeltaCatalog_AssetsNotRegistered_Failure)
#endif // AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
{
QStringList fileList = CreateArchiveFileList();
CreateArchiveFolder(GetArchiveFolderName(), fileList);
AZ_TEST_START_TRACE_SUPPRESSION;
bool createResult = CreateArchive();
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_EQ(createResult, true);
bool catalogCreated{ true };
AZ::Test::AssertAbsorber assertAbsorber;
AzToolsFramework::AssetBundleCommandsBus::BroadcastResult(catalogCreated, &AzToolsFramework::AssetBundleCommandsBus::Events::CreateDeltaCatalog, GetArchivePath().toStdString().c_str(), true);
AzToolsFramework::AssetBundleCommandsBus::BroadcastResult(catalogCreated,
&AzToolsFramework::AssetBundleCommandsBus::Events::CreateDeltaCatalog, GetArchivePath().toUtf8().constData(), true);
EXPECT_EQ(catalogCreated, false);
}
#if AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
TEST_F(ArchiveTest, DISABLED_CreateDeltaCatalog_ArchiveWithoutCatalogAssetsRegistered_Success)
TEST_F(ArchiveComponentTest, DISABLED_AddFilesToArchive_FromListFile_Success)
#else
TEST_F(ArchiveTest, CreateDeltaCatalog_ArchiveWithoutCatalogAssetsRegistered_Success)
TEST_F(ArchiveComponentTest, AddFilesToArchive_FromListFile_Success)
#endif // AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
{
QString listFile = CreateArchiveListTextFile();
CreateArchiveFolder(GetArchiveFolderName(), CreateArchiveFileList());
AZ_TEST_START_TRACE_SUPPRESSION;
std::future<bool> addResult;
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(
addResult, &AzToolsFramework::ArchiveCommandsBus::Events::AddFilesToArchive, GetArchivePath().toUtf8().constData(),
GetArchiveFolder().toUtf8().constData(), listFile.toUtf8().constData());
bool result = addResult.get();
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_TRUE(result);
}
#if AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
TEST_F(ArchiveComponentTest, DISABLED_ExtractArchive_AllFiles_Success)
#else
TEST_F(ArchiveComponentTest, ExtractArchive_AllFiles_Success)
#endif // AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
{
CreateArchiveFolder();
AZ_TEST_START_TRACE_SUPPRESSION;
bool createResult = CreateArchive();
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_TRUE(createResult);
AZ_TEST_START_TRACE_SUPPRESSION;
std::future<bool> extractResult;
AzToolsFramework::ArchiveCommandsBus::BroadcastResult(
extractResult, &AzToolsFramework::ArchiveCommandsBus::Events::ExtractArchive, GetArchivePath().toUtf8().constData(),
GetExtractFolder().toUtf8().constData());
bool result = extractResult.get();
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_TRUE(result);
QStringList archiveFiles = CreateArchiveFileList();
for (const auto& file : archiveFiles)
{
QString fullFilePath = QDir(GetExtractFolder()).absoluteFilePath(file);
QFileInfo fi(fullFilePath);
EXPECT_TRUE(fi.exists());
}
}
#if AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
TEST_F(ArchiveComponentTest, DISABLED_CreateDeltaCatalog_ArchiveWithoutCatalogAssetsRegistered_Success)
#else
TEST_F(ArchiveComponentTest, CreateDeltaCatalog_ArchiveWithoutCatalogAssetsRegistered_Success)
#endif // AZ_TRAIT_DISABLE_FAILED_ARCHIVE_TESTS
{
QStringList fileList = CreateArchiveFileList();
CreateArchiveFolder(GetArchiveFolderName(), fileList);
AZ_TEST_START_TRACE_SUPPRESSION;
bool createResult = CreateArchive();
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT;
EXPECT_EQ(createResult, true);
for (const auto& thisPath : fileList)
{
AZ::Data::AssetInfo newInfo;
newInfo.m_relativePath = thisPath.toStdString().c_str();
newInfo.m_relativePath = thisPath.toUtf8().constData();
newInfo.m_assetType = AZ::Uuid::CreateRandom();
newInfo.m_sizeBytes = 100; // Arbitrary
AZ::Data::AssetId generatedID(AZ::Uuid::CreateRandom());
@@ -212,7 +294,7 @@ namespace UnitTest
bool catalogCreated{ false };
AZ_TEST_START_TRACE_SUPPRESSION;
AzToolsFramework::AssetBundleCommandsBus::BroadcastResult(catalogCreated, &AzToolsFramework::AssetBundleCommandsBus::Events::CreateDeltaCatalog, GetArchivePath().toStdString().c_str(), true);
AzToolsFramework::AssetBundleCommandsBus::BroadcastResult(catalogCreated, &AzToolsFramework::AssetBundleCommandsBus::Events::CreateDeltaCatalog, GetArchivePath().toUtf8().constData(), true);
AZ_TEST_STOP_TRACE_SUPPRESSION_NO_COUNT; // produces different counts in different platforms
EXPECT_EQ(catalogCreated, true);
}