* Move modtime scanning tests out of APM tests file and into its own file.
Changes were kept to a minimum to get things compiling, this is just a move of code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix rebase compile errors
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Verify dependency fingerprints during Zero Analysis.
This fixes an issue where dependencies that weren't finished processing when AP shuts down would not resume when AP is started back up due to Zero Analysis ignoring dependencies when determining files to skip.
Added unit test for verification
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix compile error, make 17 a constexpr
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix compile error
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Move modtime scanning tests out of APM tests file and into its own file.
Changes were kept to a minimum to get things compiling, this is just a move of code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix rebase compile errors
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
Replaced a variable with the name "interface" to avoid conflict with MSVC keyword
Signed-off-by: AMZN-ScottR <24445312+AMZN-ScottR@users.noreply.github.com>
* Initial pass at optimizing product path dependency resolution
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add version of StripAssetPlatform that doesn't allocate or copy strings. Re-add missing test and fix up compile errors. Add benchmark test
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Change UpdateProductDependencies to directly call s_InsertProductDependencyQuery.BindAndStep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add test for same filename on multiple platforms
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Rework search logic to keep track of the source of a search path (source vs product) and keep track of which search matches which dependency to avoid doing another search through every product later on
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up code, expand test
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix paths not being lowercased by SanitizeForDatabase. Fix UpdateProductDependencies not updating existing dependencies
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add test for duplicate dependency matches. Fix saving duplicates
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Separate test into test and benchmark versions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Cleanup include
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix includes, switch hardcoded job manager setup to use JobManagerComponent instead
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Replaced wildcard_match with PathView::Match. Changed StripAssetPlatformNoCopy to use TokenizeNext. Removed Environment Create/Destroy calls. Made ScopedAllocatorFixture a base class of ScopedAllocatorSetupFixture
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add AZ Environment create/destroy on AP test environment
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add missing asserts on database functions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix incorrect usage of StripAssetPlatformNoCopy
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix source/product dependency type being ignored. Removed need for unordered_set for list of resolved dependencies. Updated unit tests
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Better variable names
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Remove testing code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix missing includes and namespaces
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Adds simple stats tracking to AssetProcessor
The system captures cumulative and individual processing times.
It avoids touching any part of the app which affects decision
making or flow, or altering any structures such as JobEntry which
are involved in processing, in order to keep it as simple and
small as possible.
Signed-off-by: lawsonamzn <70027408+lawsonamzn@users.noreply.github.com>
Leaving in the print for absorbed asserts to avoid running into future situations where important asserts are accidentally absorbed
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Enable relocation of the Project Game Release Layout
Relocating the Project Game Release Layout to another directory on the file system failed due to the querying of the engine root failing due to the ComponentApplication::m_engineRoot not using the project path stored in the SettingsRegisry if the engine root cannot be detected
Removed the ApplicationRequestBus GetEngineRoot function.
The ComponentApplicationRequestBus has a function of the same name that returns the same path.
Removed the deprecated GetAppRoot function.
The path it returns has no defined value. It was not the engine root or the project root.
Removed unused CFileUtil and CFileUtil_impl functions that were invoking the ApplicationREquestBus GetEngineRoot function.
On the way to update the functions it was discovered that they aren't called
Added a CalculateBranchToken overload that can populate a fixed_string to avoid heap allocations
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Protect against an empty list of artifacts to remove when generating the
engine.pak
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix test thread being created multiple times
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update test to not use a callback
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some more comments
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add back the callback, remove the use of a thread/sleep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
(cherry picked from commit 73202c2091)
* Fixed the return value of the ConvertToAbsolutePath function
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added the generated cmake_dependencies.*.setreg files to engine.pak (#5073)
* Copied the generated cmake_dependencies.*.setreg file to the Cache
directory
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed the platform name from the bootstrap.game.*.setreg
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Update how the project related file paths are determined when not
supplied.
The project-path determination now goes back to only detecting a "project.json" file.
It no longer attempts to detect a "Cache" directory
The project-cache-path determination now in addition to checking the
project_cache_path key searches for a "Cache" directory.
The project-path defaults to executable folder if it cannot be detected.
The copying of generated executable folder Registry directory contents
to the product cache is now removed after the archive step.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the invocation of the AssetProcessor in Jenkins to supply an
absolute path to the project.
The project-path is no longer treated as relative to the engine root,
but instead relative to the current working directory at application
startup.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added constant for the storing the name of Cache directory
Fixed typos and grammatical errors in the SettingsRegistryMergeUtils.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated UnitTest prepend the EngineRoot path to "AutomatedTesting" when
setting the project path.
This is needed now that the project-path isn't treated relative to the
EngineRoot if it is not absolute.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix AssetSeedManagerTest and PlatformAddressedAssetCatalogManagerTest
Instead of trying to used the AutomatedTesting directory as the project root, the temp directory created during the test is used as the project root.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Moved the setting of the project cache root folder and project
asset platform root folder into the `if (!projectCachePath.empty())`
block
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removing the scan up logic for the "Cache" directory.
This is no longer needed to locate the project cache path in a Project Game Release Layout.
Because the project path defaults to the executable directory if, it is not found, the Cache directory will be set to the "Cache" directory within the executable directory.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix test thread being created multiple times
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update test to not use a callback
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some more comments
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add back the callback, remove the use of a thread/sleep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Added newline to debug prints. Added additional debug prints based on previous segfault log
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add more debug messaging
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add db path to output
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix handling of absolute path dependencies when a newly added file satisfies a previously added dependency
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fixed relative path wildcard dependencies matching absolute paths
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update extra unit test to only run on windows since this problem doesn't apply to other OSes
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Pass relative path IsFileExcluded so that only paths relative to the root scan folder are matched against the exclude filters.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Revert previous change. Remove the exclude filter for the Install directory.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Pass in relative path to the exclude filter as before. Fix the AssetScanner tests.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Prepend a ./ to the relative path in order to match the exclude patterns
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Remove hack to prepend ./. Update the exclude patterns so that the hack is no longer required.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add missing ? and remove whitespace
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. IsFileExcluded() now converts the input path to a path that's relative to its corresponding scan folder.
2. Update regex patterns in gems and AutomatedTesting as well.
3. Remove unnecessary escaping for '/'.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Use ConvertToRelativePath() function to compute path relative to a scan folder.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* More fixes to regex patterns
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Remove test case which tests a hypothetical scenario that cannot occur. Fix another test case by adding scan folder.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Remove assert that's not needed since it's a valid scenario
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add unit tests for ResolveSourceFileDependencyPath
Fix crash when file is not in scan folder
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Switch to using vector of AZ strings for better error output
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Switch to using UnorderedElementsAre
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some comments and fix typo
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add an error message to AP when bad project path
Produce a log error or a dialog box error when the project path for AP
does not have a project.json and is invalid.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix a failing unit test - AssetProcessorMessages
Adding a check for 'project.json' caused BeforeRun() in a test fixture
to fail. Teardown of the fixture was also broken if the test failed to
fully startup the application manager, so added null checks there.
Added an assert to the fixture's Setup to check the status of BeforeRun().
Added additional settings registry setup to the fixture to make sure the
project path and branch token are configured before BeforeRun() is
called.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Add "Registry" folders as scan folders
For projects, Gems, and Engine, add the "Registry" folder as scan
folders.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Change params for adding scan folder
Also add trailing newlines to setreg files.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Adjust the priority order for project templates
Scan folder orders were way too high, they should be very low to become
highest priority order.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes change of default assets folder to Assets
Need to revert this change so it can be done separately.
Renamed the scan folder to 'Project/Assets' to prep for Assets folder
change later on.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Update platform configuration test to pass
Adjusted expectation of scan folder count from 1 to 2 per Gem for
'Assets' and 'Registry' now.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Update the FileIOAlias naming to make the cache, project root and engine
root paths more clear
The alias of `@root@`, `@assets@`, and `@projectplatformcache@` has been
collapsed to `@projectproductassets@`
The alias of `@devroot@` and `@engroot@` has been collapsed to
`@engroot@`
The alias of `@devassets@` and `@projectroot@` has been collapsed to
`@projectroot@`
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated use of devassets and devroot properties in python
Those properties now use projectroot and engroot
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updating the alias @engroot@ alias path comment in each platform specific LocalFileIO_*.cpp file
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed hardcoded size of 9 for the product asset alias.
The ResolvePath function now just appends the @projectproductassets@
alias with the input path
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Remove duplicate @projectproductassets@ check in ProcessFileTreeRequest
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix for typos in Hydra python test
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated LocalFileIO::Copy call on Windows to use the Unicode aware CopyFileW API
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AWSMetreicsGemAllocatorFixture to properly suppress asset
cache write errors for Test file creation.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed unneeded call to set the @projectproductasstes@ alias at the bottom of the AssetSeedManagerTest SetUp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added a deprecated alias map to the FileIO System
When a deprecated alias is accessed, the FileIO System logs an AZ_Error and indicates the alias that should be used
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated python test scripts to use the projectroot binding
Retrieving the AutomatedTesting project path based on "<devroot>/AutomatedTesting" has been removed.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated references to devroot and devgame within the codebase
The GetAbsoluteDevGameFolderPath functions has been replaced with direct call to AZ::Utils::GetProjectPath
The GetAbsoluteDevRootFolderPath functions has been replaced with direct calls to AZ::Utils::GetEnginePath
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated <engroot>/AutomatedTesting references to projectroot
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Replaced references that assumes the project path is <engroot>/AutomatedTesting with <projectroot> in the AutomatedTesting python test
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Correct casing in emfxworkspace file
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed newly added AppendPathParts function
Removed the Path constructors which accepts a PathIterable instance
The PathIterable isn't safe to return to a user of the Path class as it might be referencing temporary paths supplied via PathView arguments
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed unused parameter warning
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Undid change to the LexicallyProximate function to set the path result to the base path.
It needs to return the *this path if the pathResult is empty
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Moved the LocalFileIO ConvertToAbsolutePath implementations to AZ::Utils
Fixed the ConvertToAbsolutePath implementation for Unix platforms to use a buffer that is size PATH_MAX(4096 on all our supported Unix platforms).
Because the buffer before was AZ::IO::MaxPathLength which as a size of 1024, this was resulting in the Unix `realpath` function smashing the stack when it converts a path to an absolute path that is longer than 1024 characters
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the EditorCore.Tests to attach the AZ Environment to the EditorCore shared library that is statically loaded on launch.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed for DeprecatedAlaisesKeyVisitor Visit function causing the non string_view overloads being hidden causing a hidden base class function warning
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Changed the AWSMetricsGemMock to use a temporary for writing test assets
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the LocalFileIO::ResolvePath function to use HasRootPath to determine if a path starts with a root path such as "D:", "D:/" or "/"
IsAbsolute was not the corect check as the path of "D:" is a relative
path.
To be absolute according to the Windows the path must have a root
directory. i.e "D:/" or "D:\"
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed absolute path comment from LocalFile_UnixLike.cpp and LocalFile_Android.cpp FindFiles implementations
Updated the ConvertToAlias to supply an AZ::IO::FixedMaxPath
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Replaced usage of the @projectproductassets@ alias with @engroot@ when referring to the LmbrCentral source folder in the CopyDependencyBuilderTest and the SeedBuilderTests
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the ScriptCanvas Upgrade Tool to output backed up files to the
Project User Path instead of the engine root
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed whitespacing issues in Application.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Remove unnecessary creation of a FixedMaxPath in the UpgradeTool.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Modified testSeedFolder variable in the SeedBuilderTests to use the
@engroot@ alias instead of @projectproductassets@/.. alias when
referring to the LmbrCentral Gem source path
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated references to the Project Asset Cache in the PythonTests.
Those tests no longer use the logic `azlmbr.paths.projectroot / "Cache" / "pc"` to retrieve a path to the cache root but instead the `azlmbr.paths.projectproductassets` constant
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixed the FileIO Deprecated Alias test on Windows
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removing @projectsourceassets@ alias, as it is only used once.
Updated the PhysX EditorSystemComponent.cpp to query the ProjectPath
form the SettingsRegistry.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Replaced @projectproductassets@ alias with @products@
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Rollback changes to the PhysX EditorSystemComponent.cpp
The changes to use the ProjectPath from the SettingsRegistry has been implemented in PR #4497
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AsetProcessorManagerTest LockFileTest to check the SourceFileNotificationMessage
If the SourceFileNotificationMessage indicates that the source file was
removed, it triggers the callback to spin off a thread to remove the
product file.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Changed SourceFileNotificationMessage cast to use azrtti_cast
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix issues with seedlist for AutomatedTesting
Fixes error reporting so it will show the file hint in the tool.
Removes any missing assets from the .seed file.
Remove an unnecessary dependency from AutomatedTesting dependencies
file.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Make ArchiveComponent use AZ::IO::IArchive
Initial changes that will get the sychronous calls in ArchiveComponent
to use IArchive interface rather than external zip/7z tools.
Some of the asynchronous api are still in place, anything that wasn't
being used has been removed for now. This may change later if we move
towards all the api being asynchronous. Until then, we can't remove the
reliance upon the external archive tools completely.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Updates AZStd::thread constructors
Adds a variadic constructor which forwards args to the functor.
Because of our thread_desc extension, there was confusion on the
arugments, so the args were reordered to take the thread_desc first,
before the functor and args.
Also the thread_desc is taken as reference rather than by pointer.
Update callsites to account for this change.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Async operation of ArchiveComponent api
This sets up the ArchiveComponent to operate asynchronously. It uses
promise/future to transfer results to caller.
This is still broken, there's a few things that need to get fixed up,
but this is a good checkpoint for the work as it solidifies the api,
cleans up a bunch of unused code, and compiles.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes the platform-specific ArchiveComponen
These are no longer needed, as they control the direct interaction with
host OS tools like 7za.exe or /bin/zip.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes the platform-specific files from cmake
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Removes the 7za.exe (and legal notice)
This tool is no longer needed in the repo.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fixes usage of IArchive::GetFullPath()
This changed to return a PathView, updated to reflect that.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix promises and threads
Make sure promises are only set exactly once. This meant reworking some
of the initial error checking.
Detach threads when created. Adds [[nodiscard]] to the functions that
return a future. Since threads are detached, the future is the main way
to get communication from the thread.
Clean up interface, add comments.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* More edits to thread creation
Changes to thread construction to account for parameter change.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix some remaining issues with ArchiveComponent
Put created threads inside a container, then join them at Deactivate.
Fix asset bundler case when injecting a file with no working directory.
Fix thread constructor that applies args to a function.
Fix lambdas to take string args by value rather than reference.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fixes some remaining bugs in ArchiveComponent
Open archive as read-only during extract & list operations.
Fixes paths issues.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix initialize of opaque thread handle in thread_UnixLike.h
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed unused variable in AssetBundleComponent.cpp to fix compiler warning
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix some issues with archives
File paths in the CDR and the local headers need to match, but there
were issues with path separators and case that made it possible to get
invalid headers errors in some archives.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Adds some new ArchiveComponent unit tests
Adds new tests for extraction of archive and adding files from a file
list to an archive.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Fix file data offset issues when opening archives
When opening an INestedArchive it would run through the CDR headers to
create file entries in the zip cache. The offsets to the compressed
data were being calculated incorrectly because they were using the CDR
headers rather than jumping to the local file headers and getting
offsets from those sizes.
Removed and refactored some archive validation flow and zip cache
factory init methods to either init default or init w/ additional
validation checks.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Addresses PR feedback
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address more points of feedback in PR
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address additional PR feedback
Fixes up some error checks and uses of strings vs paths.
Enable archive component tests on Linux so we can see if they will
pass.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address PR feedback
Change the INestedArchive interface to list files as AZ::IO::Path.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Disabling the ArchiveComponent tests on Linux
They failed so we will revisit them to attempt a fix.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Rename a member variable to be more accurate
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Address feedback on PR
Bump version of Archive Components for serialize context.
Improve error messages during archive open and validation.
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
* Revert recent changes
Signed-off-by: amzn-phist <52085794+amzn-phist@users.noreply.github.com>
Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
Fixed unresolved product dependency query to be able to handle much larger queries (tested with 100,000 products) by storing queried products in a temp table first
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* fixes AssetProcessorManagerUnitTests unexpectedly fails due to writing out the branch token
* instead, the branch token is calculated ahead of time and added to the settings registry
Tests: --gtest_filter=Test/LegacyTestAdapter.*
* Final update copyright headers to reference license files at the repo root
Signed-off-by: spham <spham@amazon.com>
* Fix copyright validator unit tests to support the stale O3DE header scenario
Signed-off-by: spham <spham@amazon.com>
* Could not reproduce the unit test fail for the m_app->BeforeRun() failure,
* so I am adding file IO hooks to listen for errors during the fixture setup
Tests: failed to repro the issue
Signed-off-by: Jackson <23512001+jackalbe@users.noreply.github.com>
* Fix Delete_Real_Readonly_Fails to mark parent directory as read-only as well
Read-only files in Windows cannot be deleted. The previous version of this
code relied on that fact, and would attempt to delete a file even when
`skipReadOnly = true`, relying on the OS to refuse to delete the file if it
is read only. On Linux, it is the writable state of the *directory* that
determines if a file can be deleted or not. This fixes the test to set up
the correct situation where a file deletion would fail.
* Remove excluded items from a vector before iterating over it
Removing items from the `pathMatches` `QStringList` while iterating over it
was causing a segfault on Linux. This change separates out the item removal
from the item iteration, which allows the item iteration loop to use a
range-for loop instead of directly manipulating iterators.
* Remove invalid test that asserts a file's metadata file can have differing file casing
This test is asserting that a given source file and its accompanying
metadata file can have the the same name but differing case. This is really
testing whether or not the underlying filesystem that those files live on
is case sensitive or not. The 99% chance is that users are using the
default filesystem that their host OS gives them, NTFS on Windows, EXT* on
Linux, and APFS on Mac. Even though NTFS is case-insensitive by default,
it [can be configured per-directory](https://devblogs.microsoft.com/commandline/improved-per-directory-case-sensitivity-support-in-wsl/).
APFS as well can be configured to be case-sensitive. For users with case
sensitive filesystems, this test makes no sense. We could extend this test
to inspect the case-sensitivity of the underlying filesystem, but then it
is just testing the filesystem's behavior, which seems out of scope of this
test.
* Use a non-priviliged port for the Asset Processor tests
From https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html:
> The TCP/IP port numbers below 1024 are special in that normal users are not
> allowed to run servers on them. This is a security feaure, in that if you
> connect to a service on one of these ports you can be fairly sure that you
> have the real thing, and not a fake which some hacker has put up for you.
>
> When you run a server as a test from a non-priviliged account, you will
> normally test it on other ports, such as 2784, 5000, 8001 or 8080.
* Fix for `QDir::rmdir(".")` not working in Linux
Qt uses `::rmdir` to remove directories on Linux. This comes from
[unistd.h](https://pubs.opengroup.org/onlinepubs/007904875/functions/rmdir.html)
The documentation for that function states:
> If the path argument refers to a path whose final component is either dot
> or dot-dot, rmdir() shall fail.
So calling `dir.rmdir(".")` will never work on Linux.
Instead, get the parent directory, and remove the child directory by name.
* Avoid lowercasing source asset paths when resolving dependencies
Source asset paths may be case sensitive, so their case must be preserved
when doing operations that hit the underlying filesystem. This method was
always lowercasing them, which would cause dependencies to not be found.
* Correct test to expect product filenames to be lowercase
The modtime tests were failing in Linux due to something unrelated to file
modtime checking. The Asset Processor Manager does this during AnalyzeJob:
```
if (foundInDatabase && jobs[0].m_fingerprint == jobDetails.m_jobEntry.m_computedFingerprint)
{
// If the fingerprint hasn't changed, we won't process it.. unless...is it missing a product.
```
In this case, the test was setting up a product whose file case was the
same as the source asset, and would write it to the cache dir using mixed
case, but use the normal asset processor API to write the product file path
to the database, which recorded the path in lowercase. When the manager
then went to check if the source asset's products all exist, it checked the
lowercase path, which didn't exist.
This fixes that test failure, by updating the test to write the product
file to the cache using the proper lowercased path.
* Update test to define a "not current platform" for Linux
This test was failing because it was setting some "not current platform"
variable to be set to "pc" on Linux, when
`AssetSystem::GetHostAssetPlatform()` is defined to:
```cpp
inline const char* GetHostAssetPlatform()
{
return "mac";
return "pc";
// set this to pc because that's what bootstrap.cfg currently defines the platform to "pc", even on Linux
return "pc";
#error Unimplemented Host Asset Platform
}
```
The test would go on to assert that "pc" was simultaneously in a list and
not in the same list.
This fixes the test by updating the code to set the "not the current
platform" variable appropriately on Linux.
The expectations were also updated to improve the output on test failure.
Instead of this:
```
Value of: recogs["rend"].m_platformSpecs.contains(platformWhichIsNotCurrentPlatform)
Actual: true
Expected: false
```
You now get this:
```
Value of: recogs["rend"].m_platformSpecs.keys()
Expected: (has 3 elements and there exists some permutation of elements such that:
- element #0 is equal to pc, and
- element #1 is equal to es3, and
- element #2 is equal to server) and (doesn't contain any element that is equal to pc)
Actual: { pc, server, es3 } (of type QList<QString>), whose element #0 matches
```
* Prevent windows supported path separators to be included in the test paths for UpdateToCorrectCase_ExistingFile_ReturnsTrue_CorrectsCase
* Fix failing linux unit test "PlatformConfigurationUnitTests.TestFailReadConfigFile_RegularScanfolder"
caused by static variable not being reset from a different test run when using AssetUtilities::ComputeProjectPath
* Fix AZ_RTTI declaration for RequestEscalateAsset Message
* Implement FileWatcher for Linux to fix AssetProcessorMessages.All test (RequestAssetStatus)
* Split AssetProcessorMessages into 2 tests, one with RequestAssetStatus/ResponseAssetStatus and one without
Add The RequestAssetStatus/ResponseAssetStatus as a sandbox test because it relies on FileWatcher thread and seems to be timing related
* Remove FileWatcher_win.cpp from the Linux specific folder for FileWatcher
* - Fix build error related to non-unity builds
- Fixed failed linux test 'Test/LegacyTestAdapter.AllTests/UtilitiesUnitTest' caused by misplaced windows only EXPECT
- Remove test trait AZ_TRAIT_DISABLE_FAILED_ASSET_PROCESSOR_TESTS for linux to expose remaining failed tests
* Fixed failed linux test 'Test/LegacyTestAdapter.AllTests/RCcontrollerUnitTests' caused by misplaced windows only EXPECT
* - Fix FileWatcher unit test, disable incompatible subtests for Linux
- Fix errors in FileWatcher_linux from results of the FileWatcher Unit Test
* Remove AZ::AssetProcessor.Tests.Sandbox tests from definition and restore the original AssetProcessorMessages.All tests now that Filewatcher_linux was fixed
* Fixes for failed unit tests: AssetProcessorManagerUnitTests and AssetProcessorManagerUnitTests_JobDependencies_Fingerprint
- Caused by differences between between case-sensitive files (Linux) and non-case-sensitive Filesystems (Windows)
* Update consts in FileWatcher_linux.cpp to constexpr
* Fixes related to PR comment suggestions
* - Removed std::bind and replaced with lambda in FileWatcher_linux
- Replaced String replace functions for path separators to use AZ::IO::Path::LexicallyNormal() instead
* Restoring string replace function in PathDependencyManager::ResolveDependencies due to unit test failure
Co-authored-by: Chris Burel <burelc@amazon.com>
There are already APIs for getting a relative product path from an absolute source path, or getting a relative source path for an *existing* source file, but there were no APIs for getting a relative source path for a *new* source file. Prefabs will need this ability to be able to correctly generate a relative source path inside the prefab file before the file has been saved.
The logic for relative source paths is a little bit tricky because the paths are relative to the watch folders, and the watch folders can be nested, with different priorities to explain which should take precedence. The input paths can also include specifiers like "." and "..", which need to be reconciled before creating the final correct relative path. The included unit tests test all of the tricky edge cases that I was able to identify.
amzn-mike
Esteban Papp
Scott Romero
Nicholas Lawson
Nicholas Van Sickle
lumberyard-employee-dm
Chris Burel
SJ
amzn-phist
Artur K
sharmajs-amzn
Steve Pham
jackalbe
Aaron Ruiz Mora
Terry Michaels
Mike Balfour