birkeh/o3de
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

AP: product dependency optimization (#6619)

Browse Source 
* Initial pass at optimizing product path dependency resolution

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Add version of StripAssetPlatform that doesn't allocate or copy strings.  Re-add missing test and fix up compile errors.  Add benchmark test

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Change UpdateProductDependencies to directly call s_InsertProductDependencyQuery.BindAndStep

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Add test for same filename on multiple platforms

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Rework search logic to keep track of the source of a search path (source vs product) and keep track of which search matches which dependency to avoid doing another search through every product later on

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Clean up code, expand test

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Fix paths not being lowercased by SanitizeForDatabase.  Fix UpdateProductDependencies not updating existing dependencies

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Add test for duplicate dependency matches.  Fix saving duplicates

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Clean up code

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Separate test into test and benchmark versions

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Cleanup include

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Fix includes, switch hardcoded job manager setup to use JobManagerComponent instead

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Replaced wildcard_match with PathView::Match.  Changed StripAssetPlatformNoCopy to use TokenizeNext.  Removed Environment Create/Destroy calls.  Made ScopedAllocatorFixture a base class of ScopedAllocatorSetupFixture

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Add AZ Environment create/destroy on AP test environment

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Add missing asserts on database functions

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Fix incorrect usage of StripAssetPlatformNoCopy

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Fix source/product dependency type being ignored.  Removed need for unordered_set for list of resolved dependencies.  Updated unit tests

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Better variable names

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Remove testing code

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>

* Fix missing includes and namespaces

Signed-off-by: amzn-mike <80125227+amzn-mike@users.noreply.github.com>
This commit is contained in:
amzn-mike
2022-01-13 08:56:24 -06:00
committed by GitHub
parent 6d1a2382e8
commit fed1278fe6
14 changed files with 742 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Code/Tools/AssetProcessor/native/AssetManager/PathDependencyManager.h
+21 -2
View File
@@ -39,9 +39,27 @@ namespace AssetProcessor
PathDependencyManager(AZStd::shared_ptr<AssetDatabaseConnection> stateData, PlatformConfiguration* platformConfig);
void QueueSourceForDependencyResolution(const AzToolsFramework::AssetDatabase::SourceDatabaseEntry& sourceEntry);
void ProcessQueuedDependencyResolves();
struct SearchEntry
{
SearchEntry(AZStd::string path, bool isSourcePath, const AzToolsFramework::AssetDatabase::SourceDatabaseEntry* sourceEntry, const AzToolsFramework::AssetDatabase::ProductDatabaseEntry* productEntry)
: m_path(std::move(path)),
m_isSourcePath(isSourcePath),
m_sourceEntry(sourceEntry),
m_productEntry(productEntry) {}
AZStd::string m_path;
bool m_isSourcePath;
const AzToolsFramework::AssetDatabase::SourceDatabaseEntry* m_sourceEntry = nullptr;
const AzToolsFramework::AssetDatabase::ProductDatabaseEntry* m_productEntry = nullptr;
};
/// This function is responsible for looking up existing, unresolved dependencies that the current asset satisfies.
/// These can be dependencies on either the source asset or one of the product assets
void RetryDeferredDependencies(const AzToolsFramework::AssetDatabase::SourceDatabaseEntry& sourceEntry);
void RetryDeferredDependencies(const AzToolsFramework::AssetDatabase::SourceDatabaseEntry& sourceEntry, const AZStd::unordered_map<const SearchEntry*, AZStd::unordered_set<AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntry>>& matches, const AZStd::vector<AzToolsFramework::AssetDatabase::ProductDatabaseEntry>& products);
/// This function is responsible for taking the path dependencies output by the current asset and trying to resolve them to AssetIds
/// This does not look for dependencies that the current asset satisfies.
@@ -66,7 +84,7 @@ namespace AssetProcessor
MapSet PopulateExclusionMaps() const;
void NotifyResolvedDependencies(const AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntryContainer& dependencyContainer) const;
void SaveResolvedDependencies(const AzToolsFramework::AssetDatabase::SourceDatabaseEntry& sourceEntry, const MapSet& exclusionMaps, const AZStd::string& sourceNameWithScanFolder, const AZStd::vector<AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntry>& dependencyEntries, AZStd::string_view matchedPath, bool isSourceDependency, const AzToolsFramework::AssetDatabase::ProductDatabaseEntryContainer& matchedProducts, AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntryContainer& dependencyContainer) const;
void SaveResolvedDependencies(const AzToolsFramework::AssetDatabase::SourceDatabaseEntry& sourceEntry, const MapSet& exclusionMaps, const AZStd::string& sourceNameWithScanFolder, const AZStd::unordered_set<AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntry>& dependencyEntries, AZStd::string_view matchedPath, bool isSourceDependency, const AzToolsFramework::AssetDatabase::ProductDatabaseEntryContainer& matchedProducts, AZStd::vector<AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntry>& dependencyContainer) const;
static DependencyProductMap& SelectMap(MapSet& mapSet, bool wildcard, AzToolsFramework::AssetDatabase::ProductDependencyDatabaseEntry::DependencyType type);
/// Returns false if a path contains wildcards, true otherwise
@@ -93,5 +111,6 @@ namespace AssetProcessor
AZStd::shared_ptr<AssetDatabaseConnection> m_stateData;
PlatformConfiguration* m_platformConfig{};
DependencyResolvedCallback m_dependencyResolvedCallback{};
AZStd::vector<AzToolsFramework::AssetDatabase::SourceDatabaseEntry> m_queuedForResolve;
};
} // namespace AssetProcessor