Archive Component - Rewrite and additional work on Archive and Asset Bundler (#4332)
* Fix issues with seedlist for AutomatedTesting Fixes error reporting so it will show the file hint in the tool. Removes any missing assets from the .seed file. Remove an unnecessary dependency from AutomatedTesting dependencies file. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Make ArchiveComponent use AZ::IO::IArchive Initial changes that will get the sychronous calls in ArchiveComponent to use IArchive interface rather than external zip/7z tools. Some of the asynchronous api are still in place, anything that wasn't being used has been removed for now. This may change later if we move towards all the api being asynchronous. Until then, we can't remove the reliance upon the external archive tools completely. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Updates AZStd::thread constructors Adds a variadic constructor which forwards args to the functor. Because of our thread_desc extension, there was confusion on the arugments, so the args were reordered to take the thread_desc first, before the functor and args. Also the thread_desc is taken as reference rather than by pointer. Update callsites to account for this change. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Async operation of ArchiveComponent api This sets up the ArchiveComponent to operate asynchronously. It uses promise/future to transfer results to caller. This is still broken, there's a few things that need to get fixed up, but this is a good checkpoint for the work as it solidifies the api, cleans up a bunch of unused code, and compiles. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Removes the platform-specific ArchiveComponen These are no longer needed, as they control the direct interaction with host OS tools like 7za.exe or /bin/zip. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Removes the platform-specific files from cmake Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Removes the 7za.exe (and legal notice) This tool is no longer needed in the repo. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Fixes usage of IArchive::GetFullPath() This changed to return a PathView, updated to reflect that. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Fix promises and threads Make sure promises are only set exactly once. This meant reworking some of the initial error checking. Detach threads when created. Adds [[nodiscard]] to the functions that return a future. Since threads are detached, the future is the main way to get communication from the thread. Clean up interface, add comments. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * More edits to thread creation Changes to thread construction to account for parameter change. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Fix some remaining issues with ArchiveComponent Put created threads inside a container, then join them at Deactivate. Fix asset bundler case when injecting a file with no working directory. Fix thread constructor that applies args to a function. Fix lambdas to take string args by value rather than reference. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Fixes some remaining bugs in ArchiveComponent Open archive as read-only during extract & list operations. Fixes paths issues. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Fix initialize of opaque thread handle in thread_UnixLike.h Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com> * Removed unused variable in AssetBundleComponent.cpp to fix compiler warning Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com> * Fix some issues with archives File paths in the CDR and the local headers need to match, but there were issues with path separators and case that made it possible to get invalid headers errors in some archives. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Adds some new ArchiveComponent unit tests Adds new tests for extraction of archive and adding files from a file list to an archive. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Fix file data offset issues when opening archives When opening an INestedArchive it would run through the CDR headers to create file entries in the zip cache. The offsets to the compressed data were being calculated incorrectly because they were using the CDR headers rather than jumping to the local file headers and getting offsets from those sizes. Removed and refactored some archive validation flow and zip cache factory init methods to either init default or init w/ additional validation checks. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Addresses PR feedback Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Address more points of feedback in PR Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Address additional PR feedback Fixes up some error checks and uses of strings vs paths. Enable archive component tests on Linux so we can see if they will pass. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Address PR feedback Change the INestedArchive interface to list files as AZ::IO::Path. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Disabling the ArchiveComponent tests on Linux They failed so we will revisit them to attempt a fix. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Rename a member variable to be more accurate Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Address feedback on PR Bump version of Archive Components for serialize context. Improve error messages during archive open and validation. Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> * Revert recent changes Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com> Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
This commit is contained in:
amzn-phist
GitHub
-230
@@ -1,230 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) Contributors to the Open 3D Engine Project.
|
||||
* For complete copyright and license terms please see the LICENSE at the root of this distribution.
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*
|
||||
*/
|
||||
|
||||
#include <AzCore/std/string/string.h>
|
||||
#include <AzCore/Outcome/Outcome.h>
|
||||
#include <AzFramework/StringFunc/StringFunc.h>
|
||||
#include <AzFramework/IO/LocalFileIO.h>
|
||||
|
||||
namespace AzToolsFramework
|
||||
{
|
||||
namespace Platform
|
||||
{
|
||||
[[maybe_unused]] static const char ErrorChannel[] = "ArchiveComponent_Linux";
|
||||
|
||||
static const char ZipExePath[] = R"(/usr/bin/zip)";
|
||||
static const char UnzipExePath[] = R"(/usr/bin/unzip)";
|
||||
|
||||
static const char CreateArchiveCmd[] = "-r \"%s\" . -i *";
|
||||
|
||||
static const char ExtractArchiveCmd[] = R"(-o "%s" -d "%s")";
|
||||
|
||||
static const char AddFileCmd[] = R"("%s" "%s")";
|
||||
|
||||
static const char ExtractFileCmd[] = R"(%s "%s" %s)";
|
||||
static const char ExtractFileDestination[] = R"(%s "%s" "%s" -d "%s")";
|
||||
static const char ExtractOverwrite[] = "-o";
|
||||
static const char ExtractSkipExisting[] = "-n";
|
||||
static const char ListFilesInArchiveCmd[] = "-l %s";
|
||||
|
||||
AZStd::string GetZipExePath()
|
||||
{
|
||||
return ZipExePath;
|
||||
}
|
||||
|
||||
AZStd::string GetUnzipExePath()
|
||||
{
|
||||
return UnzipExePath;
|
||||
}
|
||||
|
||||
AZ::Outcome<AZStd::string, AZStd::string> MakePath(const AZStd::string& path)
|
||||
{
|
||||
// Create the folder if it does not already exist
|
||||
if (!AZ::IO::FileIOBase::GetInstance()->Exists(path.c_str()))
|
||||
{
|
||||
auto result = AZ::IO::FileIOBase::GetInstance()->CreatePath(path.c_str());
|
||||
if (!result)
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Path creation failed. Input path: %s \n", path.c_str()));
|
||||
}
|
||||
}
|
||||
|
||||
return AZ::Success(path);
|
||||
}
|
||||
|
||||
AZ::Outcome<AZStd::string, AZStd::string> MakeCreateArchivePath(const AZStd::string& archivePath)
|
||||
{
|
||||
// Remove the file name from the input path
|
||||
// /some/folder/path/archive.zip -> /some/folder/path/
|
||||
AZStd::string strippedArchivePath = archivePath;
|
||||
AzFramework::StringFunc::Path::StripFullName(strippedArchivePath);
|
||||
|
||||
if (strippedArchivePath.empty())
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Stripped path name is empty. Cancelling path creation. Input path: %s\n", archivePath.c_str()));
|
||||
}
|
||||
|
||||
return MakePath(strippedArchivePath);
|
||||
}
|
||||
|
||||
AZ::Outcome<AZStd::string, AZStd::string> MakeExtractArchivePath(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
|
||||
{
|
||||
if(!includeRoot)
|
||||
{
|
||||
// Create the folder for the input destination path with no modifications
|
||||
// /path/to/destination/
|
||||
return MakePath(destinationPath);
|
||||
}
|
||||
|
||||
// Get the name of the input archive. This will be the name of the root folder for the archive extraction
|
||||
// /some/folder/path/archive.zip -> archive
|
||||
AZStd::string zipFileName;
|
||||
bool result = AzFramework::StringFunc::Path::GetFileName(archivePath.c_str(), zipFileName);
|
||||
if(!result)
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Failed to get name of zip file from the archive path. Cancelling path creation. \n Input Archive Path: %s \n", archivePath.c_str()));
|
||||
}
|
||||
|
||||
// Append the root folder name to the end of the destination path
|
||||
// /path/to/destination/ + archive -> /path/to/destination/archive
|
||||
AZStd::string destinationPathWithRoot;
|
||||
result = AzFramework::StringFunc::Path::Join(destinationPath.c_str(), zipFileName.c_str(), destinationPathWithRoot);
|
||||
if(!result)
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Failed to append zip file name to the destination path. Cancelling path creation. \n Destination Path: %s \n Zip file name: %s \n", destinationPath.c_str(), zipFileName.c_str()));
|
||||
}
|
||||
|
||||
// Append a separator so that it is formatted like a folder
|
||||
// /path/to/destination/archive -> /path/to/destination/archive/
|
||||
AzFramework::StringFunc::Path::AppendSeparator(destinationPathWithRoot);
|
||||
return MakePath(destinationPathWithRoot);
|
||||
}
|
||||
|
||||
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
|
||||
{
|
||||
auto pathCreationResult = MakeCreateArchivePath(archivePath);
|
||||
if (!pathCreationResult)
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, "%s", pathCreationResult.GetError().c_str());
|
||||
return "";
|
||||
}
|
||||
AZ_UNUSED(dirToArchive);
|
||||
return AZStd::string::format(CreateArchiveCmd, archivePath.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
|
||||
{
|
||||
auto pathCreationResult = MakeExtractArchivePath(archivePath, destinationPath, includeRoot);
|
||||
if (!pathCreationResult)
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, "%s", pathCreationResult.GetError().c_str());
|
||||
return "";
|
||||
}
|
||||
|
||||
return AZStd::string::format(ExtractArchiveCmd, archivePath.c_str(), pathCreationResult.GetValue().c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& /*archivePath*/, const AZStd::string& /*listFilePath*/)
|
||||
{
|
||||
// Adding files into a archive using a list file is not currently supported
|
||||
return {};
|
||||
}
|
||||
|
||||
bool IsAddFilesToArchiveCommandSupported()
|
||||
{
|
||||
// Adding files into a archive using a list file is not currently supported
|
||||
return false;
|
||||
}
|
||||
|
||||
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file)
|
||||
{
|
||||
if (!MakeCreateArchivePath(archivePath).IsSuccess())
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, "Unable to make path for ( %s ).\n", archivePath.c_str());
|
||||
return {};
|
||||
}
|
||||
return AZStd::string::format(AddFileCmd, archivePath.c_str(), file.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
|
||||
{
|
||||
AZStd::string commandLineArgs;
|
||||
if (destinationPath.empty())
|
||||
{
|
||||
// Extract file in archive from archive path to the current directory, overwriting a file of the same name that exists there.
|
||||
commandLineArgs = AZStd::string::format(ExtractFileCmd, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str());
|
||||
}
|
||||
else
|
||||
{
|
||||
if (!MakePath(destinationPath).IsSuccess())
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, "Unable to make path ( %s ).\n", destinationPath.c_str());
|
||||
return {};
|
||||
}
|
||||
// Extract file in archive from archive path to destinationPath, overwriting a file of the same name that exists there.
|
||||
commandLineArgs = AZStd::string::format(ExtractFileDestination, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str(), destinationPath.c_str());
|
||||
}
|
||||
|
||||
return commandLineArgs;
|
||||
}
|
||||
|
||||
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath)
|
||||
{
|
||||
AZStd::string commandLineArgs = AZStd::string::format(ListFilesInArchiveCmd, archivePath.c_str());
|
||||
return commandLineArgs;
|
||||
}
|
||||
|
||||
/*
|
||||
Sample Console Output of the unzip list command
|
||||
|
||||
Archive: /var/folders/1q/12nyzqc913qgm532y2c98mnm6w4_qv/T/ArchiveTests-ra8oMy/TestArchive.pak
|
||||
Length Date Time Name
|
||||
--------- ---------- ----- ----
|
||||
0 10-14-2019 15:22 testfolder/
|
||||
1 10-14-2019 15:22 testfolder/folderfile.txt
|
||||
1 10-14-2019 15:22 basicfile.txt
|
||||
1 10-14-2019 15:22 basicfile2.txt
|
||||
0 10-14-2019 15:22 testfolder2/
|
||||
1 10-14-2019 15:22 testfolder2/sharedfolderfile2.txt
|
||||
1 10-14-2019 15:22 testfolder2/sharedfolderfile.txt
|
||||
0 10-14-2019 15:22 testfolder3/
|
||||
0 10-14-2019 15:22 testfolder3/testfolder4/
|
||||
1 10-14-2019 15:22 testfolder3/testfolder4/depthfile.bat
|
||||
--------- -------
|
||||
6 10 files
|
||||
*/
|
||||
|
||||
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries)
|
||||
{
|
||||
AZStd::vector<AZStd::string> fileEntryData;
|
||||
AzFramework::StringFunc::Tokenize(consoleOutput.c_str(), fileEntryData, "\n");
|
||||
int startingLineIdx = 3; // first line that might contain the file name
|
||||
for (size_t lineIdx = startingLineIdx; lineIdx < fileEntryData.size(); ++lineIdx)
|
||||
{
|
||||
AZStd::string& line = fileEntryData[lineIdx];
|
||||
AZStd::vector<AZStd::string> lineEntryData;
|
||||
AzFramework::StringFunc::Tokenize(line.c_str(), lineEntryData, " ");
|
||||
AZStd::string& fileName = lineEntryData.back();
|
||||
|
||||
if(fileName.back() == AZ_CORRECT_FILESYSTEM_SEPARATOR)
|
||||
{
|
||||
// if the filename ends with a separator
|
||||
// than it indicates that this is a directory
|
||||
continue;
|
||||
}
|
||||
|
||||
if(fileName.compare("-------") == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
fileEntries.emplace_back(fileName);
|
||||
}
|
||||
}
|
||||
} // namespace Platform
|
||||
} // namespace AzToolsFramework
|
||||
@@ -7,5 +7,4 @@
|
||||
#
|
||||
|
||||
set(FILES
|
||||
AzToolsFramework/Archive/ArchiveComponent_Linux.cpp
|
||||
)
|
||||
|
||||
-263
@@ -1,263 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) Contributors to the Open 3D Engine Project.
|
||||
* For complete copyright and license terms please see the LICENSE at the root of this distribution.
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*
|
||||
*/
|
||||
|
||||
#include <AzCore/std/string/string.h>
|
||||
#include <AzCore/Outcome/Outcome.h>
|
||||
#include <AzFramework/StringFunc/StringFunc.h>
|
||||
#include <AzFramework/IO/LocalFileIO.h>
|
||||
|
||||
namespace AzToolsFramework
|
||||
{
|
||||
namespace Platform
|
||||
{
|
||||
const char ErrorChannel[] = "ArchiveComponent_OSX";
|
||||
|
||||
const char ZipExePath[] = R"(/usr/bin/zip)";
|
||||
const char UnzipExePath[] = R"(/usr/bin/unzip)";
|
||||
|
||||
// v Requires investigation, the correct cmd should be R"(-r "%s" "%s/")" but tests fail
|
||||
const char CreateArchiveCmd[] = R"(-r "%s" .)";
|
||||
|
||||
const char ExtractArchiveCmd[] = R"(-o "%s" -d "%s")";
|
||||
|
||||
const char AddFileCmd[] = R"("%s" "%s" -X)";
|
||||
const char AddFilesCmd[] = R"("%s" -X %s)";
|
||||
|
||||
const char ExtractFileCmd[] = R"(%s "%s" %s)";
|
||||
const char ExtractFileDestination[] = R"(%s "%s" "%s" -d "%s")";
|
||||
const char ExtractOverwrite[] = "-o";
|
||||
const char ExtractSkipExisting[] = "-n";
|
||||
const char ListFilesInArchiveCmd[] = "-l %s";
|
||||
|
||||
AZStd::string GetZipExePath()
|
||||
{
|
||||
return ZipExePath;
|
||||
}
|
||||
|
||||
AZStd::string GetUnzipExePath()
|
||||
{
|
||||
return UnzipExePath;
|
||||
}
|
||||
|
||||
AZ::Outcome<AZStd::string, AZStd::string> MakePath(const AZStd::string& path)
|
||||
{
|
||||
// Create the folder if it does not already exist
|
||||
if (!AZ::IO::FileIOBase::GetInstance()->Exists(path.c_str()))
|
||||
{
|
||||
auto result = AZ::IO::FileIOBase::GetInstance()->CreatePath(path.c_str());
|
||||
if (!result)
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Path creation failed. Input path: %s \n", path.c_str()));
|
||||
}
|
||||
}
|
||||
|
||||
return AZ::Success(path);
|
||||
}
|
||||
|
||||
AZ::Outcome<AZStd::string, AZStd::string> MakeCreateArchivePath(const AZStd::string& archivePath)
|
||||
{
|
||||
// Remove the file name from the input path
|
||||
// /some/folder/path/archive.zip -> /some/folder/path/
|
||||
AZStd::string strippedArchivePath = archivePath;
|
||||
AzFramework::StringFunc::Path::StripFullName(strippedArchivePath);
|
||||
|
||||
if (strippedArchivePath.empty())
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Stripped path name is empty. Cancelling path creation. Input path: %s\n", archivePath.c_str()));
|
||||
}
|
||||
|
||||
return MakePath(strippedArchivePath);
|
||||
}
|
||||
|
||||
AZ::Outcome<AZStd::string, AZStd::string> MakeExtractArchivePath(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
|
||||
{
|
||||
if(!includeRoot)
|
||||
{
|
||||
// Create the folder for the input destination path with no modifications
|
||||
// /path/to/destination/
|
||||
return MakePath(destinationPath);
|
||||
}
|
||||
|
||||
// Get the name of the input archive. This will be the name of the root folder for the archive extraction
|
||||
// /some/folder/path/archive.zip -> archive
|
||||
AZStd::string zipFileName;
|
||||
bool result = AzFramework::StringFunc::Path::GetFileName(archivePath.c_str(), zipFileName);
|
||||
if(!result)
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Failed to get name of zip file from the archive path. Cancelling path creation. \n Input Archive Path: %s \n", archivePath.c_str()));
|
||||
}
|
||||
|
||||
// Append the root folder name to the end of the destination path
|
||||
// /path/to/destination/ + archive -> /path/to/destination/archive
|
||||
AZStd::string destinationPathWithRoot;
|
||||
result = AzFramework::StringFunc::Path::Join(destinationPath.c_str(), zipFileName.c_str(), destinationPathWithRoot);
|
||||
if(!result)
|
||||
{
|
||||
return AZ::Failure(AZStd::string::format("Failed to append zip file name to the destination path. Cancelling path creation. \n Destination Path: %s \n Zip file name: %s \n", destinationPath.c_str(), zipFileName.c_str()));
|
||||
}
|
||||
|
||||
// Append a separator so that it is formatted like a folder
|
||||
// /path/to/destination/archive -> /path/to/destination/archive/
|
||||
AzFramework::StringFunc::Path::AppendSeparator(destinationPathWithRoot);
|
||||
return MakePath(destinationPathWithRoot);
|
||||
}
|
||||
|
||||
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
|
||||
{
|
||||
auto pathCreationResult = MakeCreateArchivePath(archivePath);
|
||||
if (!pathCreationResult.IsSuccess())
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
|
||||
return "";
|
||||
}
|
||||
|
||||
// LY-116692. Requires proper investigation, the correct format should be:
|
||||
// AZStd::string::format(CreateArchiveCmd, archivePath.c_str(), dirToArchive.c_str());
|
||||
// but unit test ArchiveTest.ListFilesInArchiveBlocking_FilesAtThreeDepths_FilesFound fails
|
||||
AZ_UNUSED(dirToArchive);
|
||||
return AZStd::string::format(CreateArchiveCmd, archivePath.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
|
||||
{
|
||||
auto pathCreationResult = MakeExtractArchivePath(archivePath, destinationPath, includeRoot);
|
||||
if (!pathCreationResult)
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
|
||||
return "";
|
||||
}
|
||||
|
||||
return AZStd::string::format(ExtractArchiveCmd, archivePath.c_str(), pathCreationResult.GetValue().c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& listFilePath)
|
||||
{
|
||||
auto pathCreationResult = MakeCreateArchivePath(archivePath);
|
||||
if (!pathCreationResult)
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
|
||||
return "";
|
||||
}
|
||||
AZStd::string fileListStr;
|
||||
|
||||
{
|
||||
AZ::IO::FileIOStream fileStream(listFilePath.c_str(), AZ::IO::OpenMode::ModeRead | AZ::IO::OpenMode::ModeText);
|
||||
if (fileStream.IsOpen())
|
||||
{
|
||||
AZ::IO::SizeType length = fileStream.GetLength();
|
||||
AZStd::vector<char> charBuffer;
|
||||
charBuffer.resize_no_construct(length + 1);
|
||||
|
||||
fileStream.Read(length, charBuffer.data());
|
||||
charBuffer.back() = 0;
|
||||
|
||||
fileListStr.append("\"");
|
||||
fileListStr.insert(1, charBuffer.data());
|
||||
AzFramework::StringFunc::Replace(fileListStr, "\n", "\" \"");
|
||||
fileListStr.append("\"");
|
||||
}
|
||||
else
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, "Unable to read list file ( %s ) \n", listFilePath.c_str());
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
return AZStd::string::format(AddFilesCmd, archivePath.c_str(), fileListStr.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file)
|
||||
{
|
||||
auto pathCreationResult = MakeCreateArchivePath(archivePath);
|
||||
if (!pathCreationResult)
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, pathCreationResult.GetError().c_str());
|
||||
return "";
|
||||
}
|
||||
|
||||
return AZStd::string::format(AddFileCmd, archivePath.c_str(), file.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
|
||||
{
|
||||
AZStd::string commandLineArgs;
|
||||
if (destinationPath.empty())
|
||||
{
|
||||
// Extract file in archive from archive path to the current directory, overwriting a file of the same name that exists there.
|
||||
commandLineArgs = AZStd::string::format(ExtractFileCmd, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str());
|
||||
}
|
||||
else
|
||||
{
|
||||
if (!MakePath(destinationPath).IsSuccess())
|
||||
{
|
||||
AZ_Error(ErrorChannel, false, "Unable to make path ( %s ).\n", destinationPath.c_str());
|
||||
return "";
|
||||
}
|
||||
// Extract file in archive from archive path to destinationPath, overwriting a file of the same name that exists there.
|
||||
commandLineArgs = AZStd::string::format(ExtractFileDestination, overWrite ? ExtractOverwrite : ExtractSkipExisting, archivePath.c_str(), fileInArchive.c_str(), destinationPath.c_str());
|
||||
}
|
||||
|
||||
return commandLineArgs;
|
||||
}
|
||||
|
||||
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath)
|
||||
{
|
||||
AZStd::string commandLineArgs = AZStd::string::format(ListFilesInArchiveCmd, archivePath.c_str());
|
||||
return commandLineArgs;
|
||||
}
|
||||
|
||||
/*
|
||||
Sample Console Output of the unzip list command
|
||||
|
||||
Archive: /var/folders/1q/12nyzqc913qgm532y2c98mnm6w4_qv/T/ArchiveTests-ra8oMy/TestArchive.pak
|
||||
Length Date Time Name
|
||||
--------- ---------- ----- ----
|
||||
0 10-14-2019 15:22 testfolder/
|
||||
1 10-14-2019 15:22 testfolder/folderfile.txt
|
||||
1 10-14-2019 15:22 basicfile.txt
|
||||
1 10-14-2019 15:22 basicfile2.txt
|
||||
0 10-14-2019 15:22 testfolder2/
|
||||
1 10-14-2019 15:22 testfolder2/sharedfolderfile2.txt
|
||||
1 10-14-2019 15:22 testfolder2/sharedfolderfile.txt
|
||||
0 10-14-2019 15:22 testfolder3/
|
||||
0 10-14-2019 15:22 testfolder3/testfolder4/
|
||||
1 10-14-2019 15:22 testfolder3/testfolder4/depthfile.bat
|
||||
--------- -------
|
||||
6 10 files
|
||||
*/
|
||||
|
||||
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries)
|
||||
{
|
||||
AZStd::vector<AZStd::string> fileEntryData;
|
||||
AzFramework::StringFunc::Tokenize(consoleOutput.c_str(), fileEntryData, "\n");
|
||||
int startingLineIdx = 3; // first line that might contain the file name
|
||||
for (size_t lineIdx = startingLineIdx; lineIdx < fileEntryData.size(); ++lineIdx)
|
||||
{
|
||||
AZStd::string& line = fileEntryData[lineIdx];
|
||||
AZStd::vector<AZStd::string> lineEntryData;
|
||||
AzFramework::StringFunc::Tokenize(line.c_str(), lineEntryData, " ");
|
||||
AZStd::string& fileName = lineEntryData.back();
|
||||
|
||||
if(fileName.back() == AZ_CORRECT_FILESYSTEM_SEPARATOR)
|
||||
{
|
||||
// if the filename ends with a separator
|
||||
// than it indicates that this is a directory
|
||||
continue;
|
||||
}
|
||||
|
||||
if(fileName.compare("-------") == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
fileEntries.emplace_back(fileName);
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace Platform
|
||||
} // namespace AzToolsFramework
|
||||
@@ -7,5 +7,4 @@
|
||||
#
|
||||
|
||||
set(FILES
|
||||
AzToolsFramework/Archive/ArchiveComponent_Mac.cpp
|
||||
)
|
||||
|
||||
-167
@@ -1,167 +0,0 @@
|
||||
/*
|
||||
* Copyright (c) Contributors to the Open 3D Engine Project.
|
||||
* For complete copyright and license terms please see the LICENSE at the root of this distribution.
|
||||
*
|
||||
* SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
*
|
||||
*/
|
||||
|
||||
#include <AzCore/std/string/string.h>
|
||||
#include <AzCore/Component/ComponentApplicationBus.h>
|
||||
#include <AzFramework/StringFunc/StringFunc.h>
|
||||
#include <AzToolsFramework/Archive/ArchiveComponent.h>
|
||||
|
||||
namespace AzToolsFramework
|
||||
{
|
||||
namespace Platform
|
||||
{
|
||||
const char CreateArchiveCmd[] = R"(a -tzip -mx=1 "%s" -r "%s\*")";
|
||||
|
||||
// -aos is for skipping extract on existing files
|
||||
const char ExtractArchiveCmd[] = R"(x -mmt=off "%s" -o"%s\*" -aos)";
|
||||
const char ExtractArchiveWithoutRootCmd[] = R"(x -mmt=off "%s" -o"%s" -aos)";
|
||||
const char AddFilesCmd[] = R"(a -tzip "%s" @"%s")";
|
||||
const char AddFileCmd[] = R"(a -tzip "%s" "%s")";
|
||||
const char ExtractFileCmd[] = R"(e -mmt=off "%s" "%s" %s)";
|
||||
const char ExtractFileDestination[] = R"(e -mmt=off "%s" -o"%s" "%s" %s)";
|
||||
const char ExtractOverwrite[] = "-aoa";
|
||||
const char ExtractSkipExisting[] = "-aos";
|
||||
const char ListFilesInArchiveCmd[] = R"(l -r -slt "%s")";
|
||||
|
||||
AZStd::string Get7zExePath()
|
||||
{
|
||||
const char* rootPath = nullptr;
|
||||
AZ::ComponentApplicationBus::BroadcastResult(rootPath, &AZ::ComponentApplicationRequests::GetEngineRoot);
|
||||
AZStd::string exePath;
|
||||
AzFramework::StringFunc::Path::ConstructFull(rootPath, "Tools", "7za", ".exe", exePath);
|
||||
return exePath;
|
||||
}
|
||||
|
||||
AZStd::string GetZipExePath()
|
||||
{
|
||||
return Get7zExePath();
|
||||
}
|
||||
|
||||
AZStd::string GetUnzipExePath()
|
||||
{
|
||||
return Get7zExePath();
|
||||
}
|
||||
|
||||
AZStd::string GetCreateArchiveCommand(const AZStd::string& archivePath, const AZStd::string& dirToArchive)
|
||||
{
|
||||
return AZStd::string::format(CreateArchiveCmd, archivePath.c_str(), dirToArchive.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetExtractArchiveCommand(const AZStd::string& archivePath, const AZStd::string& destinationPath, bool includeRoot)
|
||||
{
|
||||
if (includeRoot)
|
||||
{
|
||||
// Extract archive path to destinationPath\<archiveFileName> and skipping extracting of existing files
|
||||
return AZStd::string::format(ExtractArchiveCmd, archivePath.c_str(), destinationPath.c_str());
|
||||
}
|
||||
else
|
||||
{
|
||||
// Extract archive path to destinationPath and skipping extracting of existing files
|
||||
return AZStd::string::format(ExtractArchiveWithoutRootCmd, archivePath.c_str(), destinationPath.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
AZStd::string GetAddFilesToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& listFilePath)
|
||||
{
|
||||
return AZStd::string::format(AddFilesCmd, archivePath.c_str(), listFilePath.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetAddFileToArchiveCommand(const AZStd::string& archivePath, const AZStd::string& file)
|
||||
{
|
||||
return AZStd::string::format(AddFileCmd, archivePath.c_str(), file.c_str());
|
||||
}
|
||||
|
||||
AZStd::string GetExtractFileCommand(const AZStd::string& archivePath, const AZStd::string& fileInArchive, const AZStd::string& destinationPath, bool overWrite)
|
||||
{
|
||||
AZStd::string commandLineArgs;
|
||||
if (destinationPath.empty())
|
||||
{
|
||||
// Extract file in archive from archive path to the current directory, overwriting a file of the same name that exists there.
|
||||
commandLineArgs = AZStd::string::format(ExtractFileCmd, archivePath.c_str(), fileInArchive.c_str(), overWrite ? ExtractOverwrite : ExtractSkipExisting);
|
||||
}
|
||||
else
|
||||
{
|
||||
// Extract file in archive from archive path to destinationPath, overwriting a file of the same name that exists there.
|
||||
commandLineArgs = AZStd::string::format(ExtractFileDestination, archivePath.c_str(), destinationPath.c_str(), fileInArchive.c_str(), overWrite ? ExtractOverwrite : ExtractSkipExisting);
|
||||
}
|
||||
|
||||
return commandLineArgs;
|
||||
}
|
||||
|
||||
AZStd::string GetListFilesInArchiveCommand(const AZStd::string& archivePath)
|
||||
{
|
||||
AZStd::string commandLineArgs = AZStd::string::format(ListFilesInArchiveCmd, archivePath.c_str());
|
||||
return commandLineArgs;
|
||||
}
|
||||
|
||||
/*
|
||||
File output for our list archive commands takes the following two patterns for files vs directories:
|
||||
|
||||
Path = basicfile2.txt
|
||||
Folder = -
|
||||
Size = 1
|
||||
Packed Size = 1
|
||||
Modified = 2019-03-26 18:31:10
|
||||
Created = 2019-03-26 18:31:10
|
||||
Accessed = 2019-03-26 18:31:10
|
||||
Attributes = A
|
||||
Encrypted = -
|
||||
Comment =
|
||||
CRC = 32D70693
|
||||
Method = Store
|
||||
Characteristics = NTFS
|
||||
Host OS = FAT
|
||||
Version = 10
|
||||
Volume Index = 0
|
||||
Offset = 44
|
||||
|
||||
Path = testfolder
|
||||
Folder = +
|
||||
Size = 0
|
||||
Packed Size = 0
|
||||
Modified = 2019-03-26 18:31:10
|
||||
Created = 2019-03-26 18:31:10
|
||||
Accessed = 2019-03-26 18:31:10
|
||||
Attributes = D
|
||||
Encrypted = -
|
||||
Comment =
|
||||
CRC =
|
||||
Method = Store
|
||||
Characteristics = NTFS
|
||||
Host OS = FAT
|
||||
Version = 20
|
||||
Volume Index = 0
|
||||
Offset = 89
|
||||
|
||||
*/
|
||||
|
||||
void ParseConsoleOutputFromListFilesInArchive(const AZStd::string& consoleOutput, AZStd::vector<AZStd::string>& fileEntries)
|
||||
{
|
||||
AZStd::vector<AZStd::string> fileEntryData;
|
||||
AzFramework::StringFunc::Tokenize(consoleOutput.c_str(), fileEntryData, "\r\n");
|
||||
for (size_t slotNum = 0; slotNum < fileEntryData.size(); ++slotNum)
|
||||
{
|
||||
AZStd::string& line = fileEntryData[slotNum];
|
||||
if (AzFramework::StringFunc::StartsWith(line, "Path = "))
|
||||
{
|
||||
if ((slotNum + 1) < fileEntryData.size())
|
||||
{
|
||||
// We're checking one past each entry we find for the Folder entry and skipping anything marked as a folder
|
||||
// See sample output above
|
||||
if (AzFramework::StringFunc::StartsWith(fileEntryData[slotNum + 1], "Folder = -"))
|
||||
{
|
||||
AzFramework::StringFunc::Replace(line, "Path = ", "", false, true);
|
||||
fileEntries.emplace_back(AZStd::move(line));
|
||||
slotNum++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} // namespace Platform
|
||||
} // namespace AzToolsFramework
|
||||
@@ -7,5 +7,4 @@
|
||||
#
|
||||
|
||||
set(FILES
|
||||
AzToolsFramework/Archive/ArchiveComponent_Windows.cpp
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user