* Move modtime scanning tests out of APM tests file and into its own file.
Changes were kept to a minimum to get things compiling, this is just a move of code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix rebase compile errors
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Initial pass at optimizing product path dependency resolution
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add version of StripAssetPlatform that doesn't allocate or copy strings. Re-add missing test and fix up compile errors. Add benchmark test
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Change UpdateProductDependencies to directly call s_InsertProductDependencyQuery.BindAndStep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add test for same filename on multiple platforms
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Rework search logic to keep track of the source of a search path (source vs product) and keep track of which search matches which dependency to avoid doing another search through every product later on
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up code, expand test
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix paths not being lowercased by SanitizeForDatabase. Fix UpdateProductDependencies not updating existing dependencies
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add test for duplicate dependency matches. Fix saving duplicates
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Separate test into test and benchmark versions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Cleanup include
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix includes, switch hardcoded job manager setup to use JobManagerComponent instead
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Replaced wildcard_match with PathView::Match. Changed StripAssetPlatformNoCopy to use TokenizeNext. Removed Environment Create/Destroy calls. Made ScopedAllocatorFixture a base class of ScopedAllocatorSetupFixture
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add AZ Environment create/destroy on AP test environment
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add missing asserts on database functions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix incorrect usage of StripAssetPlatformNoCopy
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix source/product dependency type being ignored. Removed need for unordered_set for list of resolved dependencies. Updated unit tests
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Better variable names
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Remove testing code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix missing includes and namespaces
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
Leaving in the print for absorbed asserts to avoid running into future situations where important asserts are accidentally absorbed
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fixed the return value of the ConvertToAbsolutePath function
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added the generated cmake_dependencies.*.setreg files to engine.pak (#5073)
* Copied the generated cmake_dependencies.*.setreg file to the Cache
directory
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed the platform name from the bootstrap.game.*.setreg
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Update how the project related file paths are determined when not
supplied.
The project-path determination now goes back to only detecting a "project.json" file.
It no longer attempts to detect a "Cache" directory
The project-cache-path determination now in addition to checking the
project_cache_path key searches for a "Cache" directory.
The project-path defaults to executable folder if it cannot be detected.
The copying of generated executable folder Registry directory contents
to the product cache is now removed after the archive step.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the invocation of the AssetProcessor in Jenkins to supply an
absolute path to the project.
The project-path is no longer treated as relative to the engine root,
but instead relative to the current working directory at application
startup.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added constant for the storing the name of Cache directory
Fixed typos and grammatical errors in the SettingsRegistryMergeUtils.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated UnitTest prepend the EngineRoot path to "AutomatedTesting" when
setting the project path.
This is needed now that the project-path isn't treated relative to the
EngineRoot if it is not absolute.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix AssetSeedManagerTest and PlatformAddressedAssetCatalogManagerTest
Instead of trying to used the AutomatedTesting directory as the project root, the temp directory created during the test is used as the project root.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Moved the setting of the project cache root folder and project
asset platform root folder into the `if (!projectCachePath.empty())`
block
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removing the scan up logic for the "Cache" directory.
This is no longer needed to locate the project cache path in a Project Game Release Layout.
Because the project path defaults to the executable directory if, it is not found, the Cache directory will be set to the "Cache" directory within the executable directory.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix test thread being created multiple times
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update test to not use a callback
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some more comments
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add back the callback, remove the use of a thread/sleep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Added newline to debug prints. Added additional debug prints based on previous segfault log
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add more debug messaging
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add db path to output
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix handling of absolute path dependencies when a newly added file satisfies a previously added dependency
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fixed relative path wildcard dependencies matching absolute paths
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update extra unit test to only run on windows since this problem doesn't apply to other OSes
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add unit tests for ResolveSourceFileDependencyPath
Fix crash when file is not in scan folder
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Switch to using vector of AZ strings for better error output
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Switch to using UnorderedElementsAre
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some comments and fix typo
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Updated the AsetProcessorManagerTest LockFileTest to check the SourceFileNotificationMessage
If the SourceFileNotificationMessage indicates that the source file was
removed, it triggers the callback to spin off a thread to remove the
product file.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Changed SourceFileNotificationMessage cast to use azrtti_cast
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Final update copyright headers to reference license files at the repo root
Signed-off-by: spham <spham@amazon.com>
* Fix copyright validator unit tests to support the stale O3DE header scenario
Signed-off-by: spham <spham@amazon.com>
* Fix Delete_Real_Readonly_Fails to mark parent directory as read-only as well
Read-only files in Windows cannot be deleted. The previous version of this
code relied on that fact, and would attempt to delete a file even when
`skipReadOnly = true`, relying on the OS to refuse to delete the file if it
is read only. On Linux, it is the writable state of the *directory* that
determines if a file can be deleted or not. This fixes the test to set up
the correct situation where a file deletion would fail.
* Remove excluded items from a vector before iterating over it
Removing items from the `pathMatches` `QStringList` while iterating over it
was causing a segfault on Linux. This change separates out the item removal
from the item iteration, which allows the item iteration loop to use a
range-for loop instead of directly manipulating iterators.
* Remove invalid test that asserts a file's metadata file can have differing file casing
This test is asserting that a given source file and its accompanying
metadata file can have the the same name but differing case. This is really
testing whether or not the underlying filesystem that those files live on
is case sensitive or not. The 99% chance is that users are using the
default filesystem that their host OS gives them, NTFS on Windows, EXT* on
Linux, and APFS on Mac. Even though NTFS is case-insensitive by default,
it [can be configured per-directory](https://devblogs.microsoft.com/commandline/improved-per-directory-case-sensitivity-support-in-wsl/).
APFS as well can be configured to be case-sensitive. For users with case
sensitive filesystems, this test makes no sense. We could extend this test
to inspect the case-sensitivity of the underlying filesystem, but then it
is just testing the filesystem's behavior, which seems out of scope of this
test.
* Use a non-priviliged port for the Asset Processor tests
From https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html:
> The TCP/IP port numbers below 1024 are special in that normal users are not
> allowed to run servers on them. This is a security feaure, in that if you
> connect to a service on one of these ports you can be fairly sure that you
> have the real thing, and not a fake which some hacker has put up for you.
>
> When you run a server as a test from a non-priviliged account, you will
> normally test it on other ports, such as 2784, 5000, 8001 or 8080.
* Fix for `QDir::rmdir(".")` not working in Linux
Qt uses `::rmdir` to remove directories on Linux. This comes from
[unistd.h](https://pubs.opengroup.org/onlinepubs/007904875/functions/rmdir.html)
The documentation for that function states:
> If the path argument refers to a path whose final component is either dot
> or dot-dot, rmdir() shall fail.
So calling `dir.rmdir(".")` will never work on Linux.
Instead, get the parent directory, and remove the child directory by name.
* Avoid lowercasing source asset paths when resolving dependencies
Source asset paths may be case sensitive, so their case must be preserved
when doing operations that hit the underlying filesystem. This method was
always lowercasing them, which would cause dependencies to not be found.
* Correct test to expect product filenames to be lowercase
The modtime tests were failing in Linux due to something unrelated to file
modtime checking. The Asset Processor Manager does this during AnalyzeJob:
```
if (foundInDatabase && jobs[0].m_fingerprint == jobDetails.m_jobEntry.m_computedFingerprint)
{
// If the fingerprint hasn't changed, we won't process it.. unless...is it missing a product.
```
In this case, the test was setting up a product whose file case was the
same as the source asset, and would write it to the cache dir using mixed
case, but use the normal asset processor API to write the product file path
to the database, which recorded the path in lowercase. When the manager
then went to check if the source asset's products all exist, it checked the
lowercase path, which didn't exist.
This fixes that test failure, by updating the test to write the product
file to the cache using the proper lowercased path.
* Update test to define a "not current platform" for Linux
This test was failing because it was setting some "not current platform"
variable to be set to "pc" on Linux, when
`AssetSystem::GetHostAssetPlatform()` is defined to:
```cpp
inline const char* GetHostAssetPlatform()
{
return "mac";
return "pc";
// set this to pc because that's what bootstrap.cfg currently defines the platform to "pc", even on Linux
return "pc";
#error Unimplemented Host Asset Platform
}
```
The test would go on to assert that "pc" was simultaneously in a list and
not in the same list.
This fixes the test by updating the code to set the "not the current
platform" variable appropriately on Linux.
The expectations were also updated to improve the output on test failure.
Instead of this:
```
Value of: recogs["rend"].m_platformSpecs.contains(platformWhichIsNotCurrentPlatform)
Actual: true
Expected: false
```
You now get this:
```
Value of: recogs["rend"].m_platformSpecs.keys()
Expected: (has 3 elements and there exists some permutation of elements such that:
- element #0 is equal to pc, and
- element #1 is equal to es3, and
- element #2 is equal to server) and (doesn't contain any element that is equal to pc)
Actual: { pc, server, es3 } (of type QList<QString>), whose element #0 matches
```
* Prevent windows supported path separators to be included in the test paths for UpdateToCorrectCase_ExistingFile_ReturnsTrue_CorrectsCase
* Fix failing linux unit test "PlatformConfigurationUnitTests.TestFailReadConfigFile_RegularScanfolder"
caused by static variable not being reset from a different test run when using AssetUtilities::ComputeProjectPath
* Fix AZ_RTTI declaration for RequestEscalateAsset Message
* Implement FileWatcher for Linux to fix AssetProcessorMessages.All test (RequestAssetStatus)
* Split AssetProcessorMessages into 2 tests, one with RequestAssetStatus/ResponseAssetStatus and one without
Add The RequestAssetStatus/ResponseAssetStatus as a sandbox test because it relies on FileWatcher thread and seems to be timing related
* Remove FileWatcher_win.cpp from the Linux specific folder for FileWatcher
* - Fix build error related to non-unity builds
- Fixed failed linux test 'Test/LegacyTestAdapter.AllTests/UtilitiesUnitTest' caused by misplaced windows only EXPECT
- Remove test trait AZ_TRAIT_DISABLE_FAILED_ASSET_PROCESSOR_TESTS for linux to expose remaining failed tests
* Fixed failed linux test 'Test/LegacyTestAdapter.AllTests/RCcontrollerUnitTests' caused by misplaced windows only EXPECT
* - Fix FileWatcher unit test, disable incompatible subtests for Linux
- Fix errors in FileWatcher_linux from results of the FileWatcher Unit Test
* Remove AZ::AssetProcessor.Tests.Sandbox tests from definition and restore the original AssetProcessorMessages.All tests now that Filewatcher_linux was fixed
* Fixes for failed unit tests: AssetProcessorManagerUnitTests and AssetProcessorManagerUnitTests_JobDependencies_Fingerprint
- Caused by differences between between case-sensitive files (Linux) and non-case-sensitive Filesystems (Windows)
* Update consts in FileWatcher_linux.cpp to constexpr
* Fixes related to PR comment suggestions
* - Removed std::bind and replaced with lambda in FileWatcher_linux
- Replaced String replace functions for path separators to use AZ::IO::Path::LexicallyNormal() instead
* Restoring string replace function in PathDependencyManager::ResolveDependencies due to unit test failure
Co-authored-by: Chris Burel <burelc@amazon.com>
* LYN-2537 Moved the Engine and Editor folder to be within the EngineAssets folder
* Fixed Documentation in bootstrap.cfg to correct the path to the user project specific registry file
* Adding a newline to the output of AssetCatalog 'Registering asset..., but type is not set' message
* Updating the AssetProcessorPlatformConfig.setreg Scan Folder to detect
the @ENGINEROOT@/EngineAssets/Engine path for engine runtime assets and
@ENGINEROOT@/EngineAssets/Editor path for engine tool assets
* Updating references to Icons and other assets to account for moving the
Engine and Editor folder under a single EngineAssets folder
* Moving the Engine Settings Registry folder from Engine/Registry -> Registry
* Removed the LY_PROJECT_CMAKE_PATH define as it is not portable to other locations. It is hard coded to the project location that was used for the CMake configuration. Furthermore it paths with backslashes within it are treated as escape characters and not a path separator
* Updated the LyTestTools asset_processor.py script to copy the exclude.filetag from the EngineAssets/Engine directory now
* Fixed Atom Shader Preprocessing when running using an External Project
* Updated the TSGenerateAction.cpp to fix the build error with using a renamed variable
* Updated the Install_Common.cmake ly_setup_others function to install the
EngineAssets directory and the each of the Gem's Assets directory while
maintaining the relative directory structure to the Engine Root
Also updated the install step to install the Registry folder at the
engine root
* Fixed the copying of the Registry folder to be in the install root, instead of under a second 'Registry' folder
* Moving the AssetProcessorPlatformConfig.setreg file over to the Registry folder
* Updated the LyTestTools and C++ code to point that the new location of
the AssetProcessorPlatformConfig.setreg file inside of the Registry
folder
* Renamed Test AssetProcessor*Config.ini files to have the .setreg extension
* Converted the AssetProcessor test setreg files from ini format to json
format using the SerializeContextTools convert-ini command
* Updated the AssetProcessor CMakeLists.txt to copy over the test setreg files to the build folder
* Updated the assetprocessor test file list to point at the renamed AsssetProcessor*Config setreg filenames
* Removed the Output Prefix code from the AssetProcessor. The complexity that it brought to the AP code is not needed, as users can replicate the behavior by just moving there assets underneath a another folder, underneath the scan folder
* Adding back support to read the AssetProcessorPlatformConfig.setreg file from the asset root. This is only needed for C++ UnitTests as they run in an environment where the accessing the Engine Settings Registry is not available
* Updating the Install_common.cmake logic to copy any "Assets" folder to
the install layout.
The Script has also been updated to copy over the "Assets" folder in the
Engine Root to the install layout instead of an "EngineAssets" folder
* Updating References to EngineAssets source asset folder in code to be the Assets source folder
* Moved the Engine Source Asset folder of 'EngineAssets' to a new folder name of 'Assets'. This is inline with the naming scheme we use for Gem asset folders
* Adding the EngineFinder.cmake to the AutomatedTesting project to allow it to work in a project centric manner
* Updating the LyTestTools copy_assets_to_project function to be able to copy assets with folders to the temporary project root
Fixed an issue in LyTestTools where the temporary log directory could have shutil.rmtree being called twice on it leading to an exception which fails an automated test
Updated the asset_procesor_gui_tests_2 AddScanFolder test to not use the
output prefix, but instead place the source asset root into a
subdirectory
* Correct the AssetProcessorPlatformConfig Scan Folders for the EngineAssets directory to point at the Assets directory
* Updated the asset procesor batch dependency test scan folder to point at the 'Assets' folder instead of 'EngineAssets'
amzn-mike
Chris Burel
lumberyard-employee-dm
Esteban Papp
sharmajs-amzn
Steve Pham
Terry Michaels
amzn-sj
alexpete