* Move modtime scanning tests out of APM tests file and into its own file.
Changes were kept to a minimum to get things compiling, this is just a move of code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix rebase compile errors
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Verify dependency fingerprints during Zero Analysis.
This fixes an issue where dependencies that weren't finished processing when AP shuts down would not resume when AP is started back up due to Zero Analysis ignoring dependencies when determining files to skip.
Added unit test for verification
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix compile error, make 17 a constexpr
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix compile error
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
This dependency was causing all of the asset processor modules to have to build gems, namely the unit tests which did not actually require the gems.
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Move modtime scanning tests out of APM tests file and into its own file.
Changes were kept to a minimum to get things compiling, this is just a move of code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix rebase compile errors
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fixed string_view compilation in GCC 10+.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* More GCC 10+ Fixes.
GCC 11 seems to have an issue with linkage regarding using a lambda as a default parameter in a function declaration.
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* GCC10+ Fix - Fixed binding to a temporary references.
> error: loop variable ‘pathName’ of type ‘const QString&’ binds to a temporary constructed from type ‘const char* const’ [-Werror=range-loop-construct]
415 | for (const QString& pathName : { "CrySystem",
Signed-off-by: lumberyard-employee-dm <56135373+lumberyard-employee-dm@users.noreply.github.com>
* Templates/Restricted upgrade/fixes:
Fixed template storage format: templates now only store true relative paths and no longer save "origin" paths and "optional" has been removed, it was never used.
Upgraded all templates to new standard
Template system now correctly handles child objects: Child objects no longer have to specify restricted they inherit from parent
Restricted now operates at the object level and makes no assumptions about parent
Restricted templates can now be combined and seperated on creation
ly_get_list_relative_filename has been deprecated for o3de_pal_dir
All Gems/Projects/Templates updated to use new code
Signed-off-by: byrcolin <byrcolin@amazon.com>
Replaced a variable with the name "interface" to avoid conflict with MSVC keyword
Signed-off-by: AMZN-ScottR <24445312+AMZN-ScottR@users.noreply.github.com>
* Initial pass at optimizing product path dependency resolution
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add version of StripAssetPlatform that doesn't allocate or copy strings. Re-add missing test and fix up compile errors. Add benchmark test
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Change UpdateProductDependencies to directly call s_InsertProductDependencyQuery.BindAndStep
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add test for same filename on multiple platforms
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Rework search logic to keep track of the source of a search path (source vs product) and keep track of which search matches which dependency to avoid doing another search through every product later on
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up code, expand test
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix paths not being lowercased by SanitizeForDatabase. Fix UpdateProductDependencies not updating existing dependencies
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add test for duplicate dependency matches. Fix saving duplicates
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Separate test into test and benchmark versions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Cleanup include
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix includes, switch hardcoded job manager setup to use JobManagerComponent instead
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Replaced wildcard_match with PathView::Match. Changed StripAssetPlatformNoCopy to use TokenizeNext. Removed Environment Create/Destroy calls. Made ScopedAllocatorFixture a base class of ScopedAllocatorSetupFixture
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add AZ Environment create/destroy on AP test environment
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add missing asserts on database functions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix incorrect usage of StripAssetPlatformNoCopy
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix source/product dependency type being ignored. Removed need for unordered_set for list of resolved dependencies. Updated unit tests
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Better variable names
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Remove testing code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix missing includes and namespaces
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
This change reworks the AssetProcessor's FileWatcher so that it only uses
one thread. This is motivated by getting better support for inotify on
Linux. The previous architecture required calling `inotify_init` once for
each directory that was being watched, and using separate inotify instances
for each watched tree. In addition, having separate threads per watched
tree is not necessary, and just consumes system resources. Each platform
supports watching multiple directories with the same platform-specific
watcher API, so each platform has been updated accordingly.
The interface to the FileWatcher class is greatly simplified. Previously,
it supported client-supplied filtering of the paths that would generate
notifications. This was done by subclassing `FolderWatchBase` and
implementing `OnFileChange`. However, only one filter was ever used, so
that filter is now hard-coded in the FileWatcher class, and the classes
driving the old filtering mechanism are removed. Users of the interface
now have a much easier time, they just call `AddFolderWatch` with the path
to watch, and only have to connect to one set of signals, instead of
separate signals per watched directory.
Signed-off-by: Chris Burel <burelc@amazon.com>
* AssetBuilder sends builder registration network message to AP
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add AP activating status message
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* First builder handles registration.
Fixed deadlock caused by AP and AssetBuilder waiting on each other when registering by moving AP builder start code to a thread
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up external builder registration
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Add thread description for builder manager idle thread
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Remove gem loading
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Clean up builder registration and remove unused functions
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Remove PostActivate call from batch application since it will be called after builders are registered
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Removal external builder dependency scanning since we no longer support builder dlls
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Fix missing bus disconnect
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Remove unused variable
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Moved AP-AssetBuilder specific types into AssetBuilder.Static library. Also removed some unused/old code
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
* Enable process and ap connection tests on linux
* Updated 'OpenProjectManager' to use new the ProcessLauncher argument type
* Add logic to double-escape escaped double quotes in arguments on windows platforms
* Updated argument for LaunchProjectManager to reflect new ProcessLauncher argument type
* Fixed unit test arguments for 'arg=value' condition
* Fix compile errors for BuilderManager and RHI.Edit\Utils.cpp
* PAL'ify the GetCommandLineParametersAsString() to handle windows specific behavior
Signed-off-by: Steve Pham <82231385+spham-amzn@users.noreply.github.com>
Co-authored-by: byrcolin <byrcolin@amazon.com>
The "begin and end" markers were removed due to a merge conflict. This restores them.
It also stops printing out sections that are empty - for example, if the AP runs without
processing anything, there will no longer be a "top 10 processed files" section.
Signed-off-by: lawsonamzn <70027408+lawsonamzn@users.noreply.github.com>
* Adds simple stats tracking to AssetProcessor
The system captures cumulative and individual processing times.
It avoids touching any part of the app which affects decision
making or flow, or altering any structures such as JobEntry which
are involved in processing, in order to keep it as simple and
small as possible.
Signed-off-by: lawsonamzn <70027408+lawsonamzn@users.noreply.github.com>
* Follow up fix for ATOM-13512
Update the file mask mapping in ImageBuilder.settings.
Change Reflectance.preset to use BC4 format since it only needs once channel.
Signed-off-by: Qing Tao <55564570+VickyAtAZ@users.noreply.github.com>
The AssetProcessor on Linux uses `inotify` to monitor for file updates.
The AssetProcessor also uses a `ProcessWatcher` to launch child
AssetBuilder processes. `ProcessWatcher` accomplishes this by calling
`fork()`, which duplicates the current process, then calling `exec()`.
The `fork()` call also appears to duplicate any inotify fds. This
results in the subprocess consuming duplicate inotify watches, which is
a limited system resource (Ubunut 20 defaults
`/proc/sys/fs/inotify/max_user_watches` to 65536). The AssetProcessor
still has issues with this max, but this should free up at least half of
the uses.
Signed-off-by: Chris Burel <burelc@amazon.com>
Changes include:
- Move config files from ImageProcessingAtom/Config/ folder to ImageProcessingAtom/Assets/Config/ folder so it can be watched as source depencies.
- Change GetSuggestedPreset function so it can return certain preset for certain file name.
- Move file mask mappings from preset to ImageBuilder.settings. But the file masks in preset can still be used.
- Changed from using project config folder's imageBuilder.Settings for override to using it as json merge patch.
- Expose GetFileHash function in AssetBuilderSDK api.
Signed-off-by: Qing Tao <55564570+VickyAtAZ@users.noreply.github.com>
* Cleaning up errors with default assets, used in bundled release builds
Signed-off-by: AMZN-stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
* Updated simple asset references to be to the product, not source assets
Signed-off-by: AMZN-stankowi <4838196+AMZN-stankowi@users.noreply.github.com>
Leaving in the print for absorbed asserts to avoid running into future situations where important asserts are accidentally absorbed
Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
amzn-mike
lumberyard-employee-dm
Esteban Papp
Chris Galvan
Daniel Edwards
AMZN-byrcolin
Scott Romero
nemerle
Benjamin Jillich
Chris Burel
Jeremy Ong
Steve Pham
Nicholas Lawson
Gene Walters
Gene Walters
puvvadar
Qing Tao
Tommy Walton
Guthrie Adams
Nicholas Van Sickle
AMZN-stankowi