Chris Santora b3eb54e9c9 Fixed shader and material "Failed to find builder dependency" errors
Fixed shader and material "Failed to find builder dependency" errors

Merge pull request #1462 from aws-lumberyard-dev/santorac/stabilization/2106/MissingDependencyWarnings-ATOM-15136

Fixed a couple builders to avoid adding job dependencies on source files that don't exist.

This removes mostly benign (but noisy) messages about "Failed to find builder dependency".

Changed AssetUtils::GetPossibleDepenencyPaths to return all possible source paths, rather than stopping when one is found. This function is now used to report a list of all possible source dependencies, so that CreateJobs will get called by the AP whenever a file shows up at one of those locations. If a file was missing before and then appears, this will cause the builders to wake up and add the appropriate job dependencies on the new files. In ShaderVariantAssetBuilder and MaterialBuilder, we now use GetPossibleDepenencyPaths to report source dependencies rather than job dependencies. We only report a job dependency when the actual source file has been identified. This should all now be consistent with the intended design of the AP's dependency systems (the prior approach was a hack based on misunderstanding of what source dependencies are).

SrgLayoutBuilder's change is a bit tricky. The above changes did not fix all of the "Failed to find builder dependency" messages because AzslBuilder sometimes skips particular files in CreateJobs. When this happens, it is invalid to report an AzslBuilder job dependency on that file. So I copied the same conditional code that is used to skip an azsli file in CreateJobs, and used that to skip AddAzslBuilderJobDependency() in SrgLayoutBuilder as well.

With all these changes combined, I do think we've solved the issue where jobs fail to evict outdated jobs, as described in ATOM-15134. However, we are not yet seeing the iteration time improvements we were hoping for. Before these changes I was seeing roughly a 0.5 minute delay for the initial change, and a 2 minute delay for a subsequent change. With these changes it's more like 0.5 mibutes and 1.5 minutes. It appears that the AP scan is being starved by all the AssetBuilder processing going on, and perhaps IO contention. I suspect that this will be greatly improved on the development branch where we no longer have AzslBuilder and SrgLayoutBuilder slowing things down.

ATOM-15136 Builder dependency errors reported in mainline
ATOM-15134 Replace GetPossibleDepenencyPaths Approach with Source Dependencies

Testing: 
- Deleted and rebuild the ASV cache a few times. 
- Ran through a few different scenarios of adding and removing referenced files in the project folder and gem folder. Covering...
  - Parent materials
  - Material types
  - Material functors
- I discovered a particular scenario that produces incorrect results, but I verified the scenario fails without my changes too: LYN-4757
- Opened a few levels in AtomTest.
2021-06-22 20:12:40 -07:00
2021-06-16 14:45:11 -07:00
2021-03-08 14:30:57 -08:00
2021-04-08 10:45:16 -07:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00
2021-03-08 14:30:57 -08:00

Project Spectra Private Preview

Confidentiality; Pre-Release Access

Welcome to the Project Spectra Private Preview. This is a confidential pre-release project; your use is subject to the nondisclosure agreement between you (or your organization) and Amazon. Do not disclose the existence of this project, your participation in it, or any of the materials provided, to any unauthorized third party. To request access for a third party, please contact Royal O'Brien, obriroya@amazon.com.

Full instructions can be found here:

https://docs.o3de.org/docs/welcome-guide/setup/setup-from-github/

(Note: Contact Royal or Doug Erickson, dougeric@amazon.com for access)

Updates to this readme

May 14, 2021

  • Removed instructions for the 3rdParty zip file and downloader URL. This is no longer a requirement.
  • Updated instructions for dependencies
  • Links to full documentation

April 7-13, 2021

  • Updates to the 3rdParty zip file

March 25, 2021

  • Initial commit for instructions

Download and Install

This repository uses Git LFS for storing large binary files. You will need to create a Github personal access token to authenticate with the LFS service.

To install Git LFS, download the binary here: https://git-lfs.github.com/.

After installation, you will need to install the necessary git hooks with this command

git lfs install

Create a Git Personal Access Token

You will need your personal access token credentials to authenticate when you clone the repository and when downloading objects from Git LFS

Create a personal access token with the 'repo' scope.

During the clone operation, you will be prompted to enter a password. Your token will be used as the password. You will also be prompted a second time for Git LFS.

Recent versions of Git install a credential manager to store your credentials so you don't have to put in the credentials for every request.

It is highly recommended you check that you have a credential manager installed and configured

For Linux and Mac, use the following commands to store credentials

Linux:

git config --global credential.helper cache

Mac:

git config --global credential.helper osxkeychain

Clone the repository

> git clone https://github.com/aws-lumberyard/o3de.git
Cloning into 'o3de'...

# initial prompt for credentials to download the repository code
# enter your Github username and personal access token

remote: Counting objects: 29619, done.
Receiving objects: 100% (29619/29619), 40.50 MiB | 881.00 KiB/s, done.
Resolving deltas: 100% (8829/8829), done.
Updating files: 100% (27037/27037), done.

# second prompt for credentials when downloading LFS files
# enter your Github username and personal access token

Filtering content: 100% (3853/3853), 621.43 MiB | 881.00 KiB/s, done.

If you have the Git credential manager core or other credential helpers installed, you should not be prompted for your credentials anymore.

Building the Engine

Build Requirements and redistributables

Windows

Optional

  • WWise - 2019.2.8.7432 minimum: https://www.audiokinetic.com/download/
    • Note: This requires registration and installation of a client to download
    • You will also need to set a environment variable: set LY_WWISE_INSTALL_PATH=<path to WWise version>
    • For example: set LY_WWISE_INSTALL_PATH="C:\Program Files (x86)\Audiokinetic\Wwise 2019.2.8.7432"

Quick Start Build Steps

  1. Create a writable folder to cache 3rd Party dependencies. You can also use this to store other redistributable SDKs.

    For the 0.5 branch - Create an empty text file named 3rdParty.txt in this folder, to allow a legacy CMake validator to pass

  2. Install the following redistributables to the following:

    • Visual Studio and VC++ redistributable can be installed to any location
    • CMake can be installed to any location, as long as it's available in the system path
    • WWise can be installed anywhere, but you will need to set an environment variable for CMake to detect it: set LY_WWISE_INSTALL_PATH=<path to WWise>
  3. Navigate into the repo folder, then download the python runtime with this command

    For the 0.5 branch - Set this environment variable prior to the get_python command below:

    set LY_PACKAGE_SERVER_URLS=https://d2c171ws20a1rv.cloudfront.net
    
    python\get_python.bat
    
  4. Configure the source into a solution using this command line, replacing and <3rdParty cache path> to a path you've created:

    cmake -B <your build path> -S <your source path> -G "Visual Studio 16" -DLY_3RDPARTY_PATH=<3rdParty cache path> -DLY_UNITY_BUILD=ON -DLY_PROJECTS=AutomatedTesting 
    

    Note: Do not use trailing slashes for the <3rdParty cache path>

  5. Alternatively, you can do this through the CMake GUI:

    1. Start cmake-gui.exe
    2. Select the local path of the repo under "Where is the source code"
    3. Select a path where to build binaries under "Where to build the binaries"
    4. Click "Configure"
    5. Wait for the key values to populate. Fill in the fields that are relevant, including LY_3RDPARTY_PATH and LY_PROJECTS
    6. Click "Generate"
  6. The configuration of the solution is complete. To build the Editor and AssetProcessor to binaries, run this command inside your repo:

    cmake --build <your build path> --target AutomatedTesting.GameLauncher AssetProcessor Editor --config profile -- /m
    
  7. This will compile after some time and binaries will be available in the build path you've specified

Setting up new projects

  1. While still within the repo folder, register the engine with this command:
    scripts\o3de.bat register --this-engine
    
  2. Setup new projects using the o3de create-project command. In the 0.5 branch, the project directory must be a subdirectory in the repo folder.
    <Repo path>\scripts\o3de.bat create-project --project-path <your new project path>
    
  3. Register the engine to the project
    <Repo path>\scripts\o3de.bat register --project-path <New project path>
    
  4. Once you're ready to build the project, run the same set of commands to configure and build:
    cmake -B <your project build path> -S <your new project source path> -G "Visual Studio 16" -DLY_3RDPARTY_PATH=<3rdParty cache path>
    
    // For the 0.5 branch, you must build a new Editor for each project:
    cmake --build <your project build path> --target <New Project Name>.GameLauncher Editor --config profile -- /m
    
    // For all other branches, just build the project:
    cmake --build <your project build path> --target <New Project Name>.GameLauncher --config profile -- /m
    

For a tutorial on project configuration, see Creating Projects Using the Command Line in the documentation.

License

For terms please see the LICENSE*.TXT file at the root of this distribution.

S
Description
Open 3D Engine (O3DE) is an Apache 2.0-licensed multi-platform 3D engine that enables developers and content creators to build AAA games, cinema-quality 3D worlds, and high-fidelity simulations without any fees or commercial obligations.
Readme 150 MiB
Languages
C++ 82.6%
Python 13.1%
CMake 1.5%
Roff 1.2%
Lua 0.5%
Other 0.8%