The AssetProcessor on Linux uses `inotify` to monitor for file updates.
The AssetProcessor also uses a `ProcessWatcher` to launch child
AssetBuilder processes. `ProcessWatcher` accomplishes this by calling
`fork()`, which duplicates the current process, then calling `exec()`.
The `fork()` call also appears to duplicate any inotify fds. This
results in the subprocess consuming duplicate inotify watches, which is
a limited system resource (Ubunut 20 defaults
`/proc/sys/fs/inotify/max_user_watches` to 65536). The AssetProcessor
still has issues with this max, but this should free up at least half of
the uses.
Signed-off-by: Chris Burel <burelc@amazon.com>
Changes include:
- Move config files from ImageProcessingAtom/Config/ folder to ImageProcessingAtom/Assets/Config/ folder so it can be watched as source depencies.
- Change GetSuggestedPreset function so it can return certain preset for certain file name.
- Move file mask mappings from preset to ImageBuilder.settings. But the file masks in preset can still be used.
- Changed from using project config folder's imageBuilder.Settings for override to using it as json merge patch.
- Expose GetFileHash function in AssetBuilderSDK api.
Signed-off-by: Qing Tao <55564570+VickyAtAZ@users.noreply.github.com>
* Add cvar for aws log level
Signed-off-by: onecent1101 <liug@amazon.com>
* Remove old crysystem registered cmd and add safeguard to get cvar from console
Signed-off-by: onecent1101 <liug@amazon.com>
* Suppress error for client auth unit test as logging was routed to warning before
Signed-off-by: onecent1101 <liug@amazon.com>
- Fix error with EngineFinder using the wrong path to locate project.json
- Simplified and expanded clang detection
- Remove forcing clang-12 for builds and will rely on the new cmake detection of clang by default
Signed-off-by: Steve Pham <82231385+spham-amzn@users.noreply.github.com>
* Cleaning up errors with default assets, used in bundled release builds
Signed-off-by: AMZN-stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Updated simple asset references to be to the product, not source assets
Signed-off-by: AMZN-stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
Leaving in the print for absorbed asserts to avoid running into future situations where important asserts are accidentally absorbed
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
- Added support for NDK r23
- Added support for the versioned Android 'cmdline-tools' package which replace the deprecated 'tools' package
- Updated max Gradle version to latest point release of 7.0 to fix volume query bug during sync in some configurations
- Updated min Android Gradle plugin version to latest point release of 4.2
- Fixed issue with Gradle task chaining for some custom copy tasks
- Replaced use of deprecated 'jcenter' repo with 'mavenCentral' in Android project generator template
- Increased debug logging around adb calls in Android deployment script
- Fixed a couple try/except cases that were looking for the wrong exception when invoking adb_* functions
- Fixed issue where asset cache would get deleted when regenerating an existing Android Gradle project with --overwrite-existing
- Fixed issues with running Android project generation scripts on Unix systems
Signed-off-by: AMZN-ScottR 24445312+AMZN-ScottR@users.noreply.github.com
* Enable relocation of the Project Game Release Layout
Relocating the Project Game Release Layout to another directory on the file system failed due to the querying of the engine root failing due to the ComponentApplication::m_engineRoot not using the project path stored in the SettingsRegisry if the engine root cannot be detected
Removed the ApplicationRequestBus GetEngineRoot function.
The ComponentApplicationRequestBus has a function of the same name that returns the same path.
Removed the deprecated GetAppRoot function.
The path it returns has no defined value. It was not the engine root or the project root.
Removed unused CFileUtil and CFileUtil_impl functions that were invoking the ApplicationREquestBus GetEngineRoot function.
On the way to update the functions it was discovered that they aren't called
Added a CalculateBranchToken overload that can populate a fixed_string to avoid heap allocations
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Protect against an empty list of artifacts to remove when generating the
engine.pak
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix test thread being created multiple times
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update test to not use a callback
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some more comments
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add back the callback, remove the use of a thread/sleep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
(cherry picked from commit 73202c2091)
* Fixed the return value of the ConvertToAbsolutePath function
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added the generated cmake_dependencies.*.setreg files to engine.pak (#5073)
* Copied the generated cmake_dependencies.*.setreg file to the Cache
directory
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed the platform name from the bootstrap.game.*.setreg
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Update how the project related file paths are determined when not
supplied.
The project-path determination now goes back to only detecting a "project.json" file.
It no longer attempts to detect a "Cache" directory
The project-cache-path determination now in addition to checking the
project_cache_path key searches for a "Cache" directory.
The project-path defaults to executable folder if it cannot be detected.
The copying of generated executable folder Registry directory contents
to the product cache is now removed after the archive step.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the invocation of the AssetProcessor in Jenkins to supply an
absolute path to the project.
The project-path is no longer treated as relative to the engine root,
but instead relative to the current working directory at application
startup.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added constant for the storing the name of Cache directory
Fixed typos and grammatical errors in the SettingsRegistryMergeUtils.cpp
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated UnitTest prepend the EngineRoot path to "AutomatedTesting" when
setting the project path.
This is needed now that the project-path isn't treated relative to the
EngineRoot if it is not absolute.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix AssetSeedManagerTest and PlatformAddressedAssetCatalogManagerTest
Instead of trying to used the AutomatedTesting directory as the project root, the temp directory created during the test is used as the project root.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Moved the setting of the project cache root folder and project
asset platform root folder into the `if (!projectCachePath.empty())`
block
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removing the scan up logic for the "Cache" directory.
This is no longer needed to locate the project cache path in a Project Game Release Layout.
Because the project path defaults to the executable directory if, it is not found, the Cache directory will be set to the "Cache" directory within the executable directory.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix test thread being created multiple times
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update test to not use a callback
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add some more comments
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add back the callback, remove the use of a thread/sleep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fixed some files missed when groundplane_521 was renamed to 512 (#4958)
* Fixed references to 521x521 to reference the correct 512x512 FBX file
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Fixed asset hints
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Moved the Asset Catalog loading from LmbrCentral to the AzFramework::Application (#4568)
* Moved the loading of the AssetCatalog from LmbrCentralSystemComponent to AzFramework Application
Modified the AssetCatalog::InitializeCatalog function to no longer rely on the TickBus to send out the `AssetCatalogEventBus::OnCatalogLoaded` event.
It now queues a function on the AssetCatalogRequestBus to send the OnCatalogLoaded event as soon as the dispatching for the AssetCatalogRequestBus has completed on the current thread.
This is done by updating the AssetCatalogRequestBus to use EBus ThreadDispatchPolicy to add a callback to invoke any queued function has soon a thread has finished dispatching and has released its DispatchMutex
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AssetCatalogRequestBus to add a custom DispatchLockGuard
The AssetCatalogRequestBus uses the custom lock guard to dispatch queued
events after it has unlocked it's context mutex for the current thread.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed GetContext call from the
AssetCatalogRequests::PostThreadDispatchInvoker
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the definition of FileTagQueryManager::GetDefaultFileTagFilePath
function to return a path
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AZ_CONSOLEFREEFUNC macro to actually use the _NAME
The _NAME parameter was not being used before, resulting in the Console
stringified name of the function being used.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed CrySystem dependencies from the BundlingSystemComponent
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Moved the loading of the AssetCatalog from LmbrCentralSystemComponent to AzFramework Application
Modified the AssetCatalog::InitializeCatalog function to no longer rely on the TickBus to send out the `AssetCatalogEventBus::OnCatalogLoaded` event.
It now queues a function on the AssetCatalogRequestBus to send the OnCatalogLoaded event as soon as the dispatching for the AssetCatalogRequestBus has completed on the current thread.
This is done by updating the AssetCatalogRequestBus to use EBus ThreadDispatchPolicy to add a callback to invoke any queued function has soon a thread has finished dispatching and has released its DispatchMutex
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AssetCatalogRequestBus to add a custom DispatchLockGuard
The AssetCatalogRequestBus uses the custom lock guard to dispatch queued
events after it has unlocked it's context mutex for the current thread.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed GetContext call from the
AssetCatalogRequests::PostThreadDispatchInvoker
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the definition of FileTagQueryManager::GetDefaultFileTagFilePath
function to return a path
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Updated the AZ_CONSOLEFREEFUNC macro to actually use the _NAME
The _NAME parameter was not being used before, resulting in the Console
stringified name of the function being used.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed CrySystem dependencies from the BundlingSystemComponent
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Addded missing template parameter to AssetCatalogRequests
The fixes the compile error.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Adding AssetBus::MultiHandler::BusDisconnect call
The BlastSystemComponent was connecting to the Bus, but not
disconnecting from it, causing an assert to fire to it being a
multi-thread bus
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added support for DataDrive lifecycle events to the ComponentApplication
The events are using the SettingsRegistry NotifyEvent to track when
certain keys are modified to signal handlers.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Corrected invalid JSON creation in ModuleManager::DeactivateEntities
Resolved clang warning about used type alias
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fix for dangling reference in lambda registered to the SettingsRegistry
Notifier event
This was causing the EditorPythonBinding tests to crash due to the
following circumstances.
First Python has created an instance of a SettingsRegistryProxy
Second the SettingsRegistry sends an event during the time when the
SettingsRegistryProxy exists.
This issue was exposed due to the ComponentApplication Lifecycle events
using the SettingsRegistry to dispatch during various times of the
application workflow.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added the generated cmake_dependencies.*.setreg files to engine.pak (#5073)
* Copied the generated cmake_dependencies.*.setreg file to the Cache
directory
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed the platform name from the bootstrap.game.*.setreg
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Fixes for release builds with DCO fix (#5164)
* This set of changes is work toward allowing release builds to work with asset bundler generated bundles and legacy, non-prefab levels. This requires some other in-flight changes before this work is complete.
Updated engine seed list + fixed automated test
ComponentApplicationLifecycle has the ability to automatically register events if asked to register a handler and the event doesn't exist. This is only intended for cases where you need to register a handler early in startup before the settings registry file is loaded.
Added two new lifecycle events: One after the system entity has been activated, and one after the system interface has been created.
If you load an archive before the system entity has been activated, archive.cpp caches information about those archives until that time, so it can finish registration. This is because the serialization system and BundlingSystemComponent both need to be available to do this registration, but the bundles have to be loaded before those are initialized so that the settings registry file can be loaded.
Fixed an error were mounted pak files were searching for levels.pak and not level.pak, and not finding them. I'm pretty sure this logic doesn't do anything functional either way, but I've been testing legacy levels with this change and they work now.
Moved wildcard pak loading to where engine.pak is loaded. This is because the settings registry file that defines the IO stack to spin up must be available early in application startup, and this file must be within a mounted pak file. If you're using asset bundler generated bundles, they need to be loaded at this time so that file can be loaded.
Atom's BootstrapSystemComponent.cpp no longer initializes on AssetCatalogLoaded, and instead initializes on the ApplicationLifecycle event SystemInterfaceCreated. This is because the base assetcatalog.xml file is really just a development time concept, this file should not be used in packaged release builds, because those builds will make use of delta catalogs in each bundle loaded. The asset catalog contains the list of all assets that were in the cache at development time, and this contains content that developers don't want to ship, and they may want to specifically hide from their customers, so data miners don't find secrets about upcoming game content.
Recovering from a branch that had incorrect DCO
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Fixed an incorrect ebus disconnect and removed an include that's no longer needed
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Fixed a copy and paste typo from trying to recover the previous pull request
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Updated product IDs for the settings registry builder to no longer collide with the JSON builder. Now they are based on a hash of the configuration.
Updated the engine default seed list to include the new asset ID info for the renamed bootstrap file
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Updated the path to the application lifecycle events, because runtime settings aren't included in the merged bootstrap file.
Addressed some feedback on printing out a string view on an error
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Removed a test that uses old assets that aren't relevant. We may not need this test anymore, but if we do we've backlogged a task to create a new test to cover this behavior without using old assets.
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Renamed SystemInterfaceCreated event to LegacySystemInterfaceCreated
Removed SystemEntityActivated event. Now that I have the rest of the fixes in this pull request, this new event wasn't needed, the already existing SystemComponentsActivated event does what I need.
Changed list to vector
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
Co-authored-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Added newline to debug prints. Added additional debug prints based on previous segfault log
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add more debug messaging
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add db path to output
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix handling of absolute path dependencies when a newly added file satisfies a previously added dependency
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fixed relative path wildcard dependencies matching absolute paths
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Update extra unit test to only run on windows since this problem doesn't apply to other OSes
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* This set of changes is work toward allowing release builds to work with asset bundler generated bundles and legacy, non-prefab levels. This requires some other in-flight changes before this work is complete.
Updated engine seed list + fixed automated test
ComponentApplicationLifecycle has the ability to automatically register events if asked to register a handler and the event doesn't exist. This is only intended for cases where you need to register a handler early in startup before the settings registry file is loaded.
Added two new lifecycle events: One after the system entity has been activated, and one after the system interface has been created.
If you load an archive before the system entity has been activated, archive.cpp caches information about those archives until that time, so it can finish registration. This is because the serialization system and BundlingSystemComponent both need to be available to do this registration, but the bundles have to be loaded before those are initialized so that the settings registry file can be loaded.
Fixed an error were mounted pak files were searching for levels.pak and not level.pak, and not finding them. I'm pretty sure this logic doesn't do anything functional either way, but I've been testing legacy levels with this change and they work now.
Moved wildcard pak loading to where engine.pak is loaded. This is because the settings registry file that defines the IO stack to spin up must be available early in application startup, and this file must be within a mounted pak file. If you're using asset bundler generated bundles, they need to be loaded at this time so that file can be loaded.
Atom's BootstrapSystemComponent.cpp no longer initializes on AssetCatalogLoaded, and instead initializes on the ApplicationLifecycle event SystemInterfaceCreated. This is because the base assetcatalog.xml file is really just a development time concept, this file should not be used in packaged release builds, because those builds will make use of delta catalogs in each bundle loaded. The asset catalog contains the list of all assets that were in the cache at development time, and this contains content that developers don't want to ship, and they may want to specifically hide from their customers, so data miners don't find secrets about upcoming game content.
Recovering from a branch that had incorrect DCO
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Fixed an incorrect ebus disconnect and removed an include that's no longer needed
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Fixed a copy and paste typo from trying to recover the previous pull request
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Updated product IDs for the settings registry builder to no longer collide with the JSON builder. Now they are based on a hash of the configuration.
Updated the engine default seed list to include the new asset ID info for the renamed bootstrap file
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Updated the path to the application lifecycle events, because runtime settings aren't included in the merged bootstrap file.
Addressed some feedback on printing out a string view on an error
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Removed a test that uses old assets that aren't relevant. We may not need this test anymore, but if we do we've backlogged a task to create a new test to cover this behavior without using old assets.
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Renamed SystemInterfaceCreated event to LegacySystemInterfaceCreated
Removed SystemEntityActivated event. Now that I have the rest of the fixes in this pull request, this new event wasn't needed, the already existing SystemComponentsActivated event does what I need.
Changed list to vector
Signed-off-by: stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Modify the AssetBundler to correctly identify the Gems that are enabled in the current active project
Signed-off-by: Tommy Walton <waltont@amazon.com>
* Removed unnecessary if() statement and updated the comment.
Signed-off-by: Tommy Walton <waltont@amazon.com>
* Disabling gem loading in the asset bundler tests, just like the asset bundler itself.
Signed-off-by: Tommy Walton <waltont@amazon.com>
* Copied the generated cmake_dependencies.*.setreg file to the Cache
directory
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Removed the platform name from the bootstrap.game.*.setreg
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Pass relative path IsFileExcluded so that only paths relative to the root scan folder are matched against the exclude filters.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Revert previous change. Remove the exclude filter for the Install directory.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Pass in relative path to the exclude filter as before. Fix the AssetScanner tests.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Prepend a ./ to the relative path in order to match the exclude patterns
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Remove hack to prepend ./. Update the exclude patterns so that the hack is no longer required.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Add missing ? and remove whitespace
Signed-off-by: amzn-sj <srikkant@amazon.com>
* 1. IsFileExcluded() now converts the input path to a path that's relative to its corresponding scan folder.
2. Update regex patterns in gems and AutomatedTesting as well.
3. Remove unnecessary escaping for '/'.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Use ConvertToRelativePath() function to compute path relative to a scan folder.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* More fixes to regex patterns
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Remove test case which tests a hypothetical scenario that cannot occur. Fix another test case by adding scan folder.
Signed-off-by: amzn-sj <srikkant@amazon.com>
* Remove assert that's not needed since it's a valid scenario
Signed-off-by: amzn-sj <srikkant@amazon.com>
AMZN-Phil
Chris Burel
nggieber
AMZN-nggieber
Qing Tao
Alex Peterson
Tommy Walton
AMZN-Phil
Vincent Liu
Guthrie Adams
Guthrie Adams
Steve Pham
Nicholas Van Sickle
AMZN-stankowi
Gene Walters
amzn-mike
Scott Romero
lumberyard-employee-dm
Tom Hulton-Harrop
Chris Galvan
amzn-sj
Esteban Papp
Mike Balfour
Allen Jackson
Adi Bar-Lev
Tommy Walton
Danilo Aimini
puvvadar