Summary:
The O3DE Template system provides several gem templates that users instantiate in order to start iteration on a new gem.
Those Gems templates need expose several CMake targets that allows a gem to build it's code into a library that is loaded by O3DE applications as a "plugin", as well as targets for sharing a gem public API with outside dependencies(other gems and libraries) as well as private targets for sharing code among the gem "plugin" and test targets.
Currently as it stands , the gem templates exposes a ${GemName}.Static
library target whose purpose is to share source files and include directories among the gem plugin targets. It is meant to be a private target not used outside of gem.
Additionally the gem templates exposed a ${GemName}
and ${GemName}.Editor
target that are a MODULE library type in non-monolithic builds and a static library in monolithic builds.
CMake enforces that MODULE library targets can't be linked into other targets can only be loaded dynamically using dlopen functionality. See CMake add_library documentation for more details.
Since monolithic builds are meant for release of a game or simulation application, all the shared libraries targets are converted to the static library targets and linked into the final application in order to prevent shared library dependency issues, reduce the size of the overall binary folder and to allow the linker to perform de-duplication and optimizations over all the code.
What is the motivation for this suggestion?
Why is this important?
Having a convention for separating a private vs public API within the gem templates allow authors of gems to enforce boundaries as to what functionality a dependents libraries should use and what functionality is internal functionality that is subject to change among versions.
Furthermore a public API convention allows users of gem to learn a recommended approach for using other how to depend on the code of other gems
What are the use cases for this suggestion?
The primary use case is for dependent gems and other libraries to be able to know which targets they can depend on as part of the public API and therefore have an expectation of gem functionality that is meant to be stable.
This also empowers authors of gem to separate their internal implementation from their public exposed API, which allows them to set user expectations of updates they make to their gems.
Gem authors would be allowed to change their internal implementation with little push back as public users wouldn't depend on that implementation. When it comes to updating the public API gem authors have to be cognizant of changes they make as to not break other gems using that public API.
This creates an unspoken contract between the gem maintainer and the gem user, that the gem maintainer is allowed to change the internal implementation in anyway they would like without external insight, while any public API changes can and should receive scrutiny from users of that gem
What should the outcome be if this suggestion is implemented?
The following Gem Templates should be updated as of 2022-04-26
CppToolGem
DefaultGem
PythonToolGem
Suggested design description:
The first update to those gem would be to add a new ${GemName}.API
and ${GemName}.Editor.API
INTERFACE target that exposes the public API of the gem.
The those targets are meant to transfer over the interface include directories, compile definitions and options, and build dependencies that any outside gem/library using that gem needs to depend on.
By default the following gem variant API aliases will be added to facilitate users depending on the Gem API of variants instead of the runtime and editor API targets.
Here is the list of new variant API aliases
${GemName}.Clients.API
${GemName}.Servers.API
${GemName}.Tools.API
${GemName}.Builders.API
The next update is to create Gem variant aliases for the API targets. This allows a gem to expose a specific API based on the Gem variants that it is used with.
For example a gem could expose a different API to the AssetProcessor than the Editor by making a separate ${GemName}.Builders.API
INTERFACE target vs ${GemName}.Editor.API
INTERFACE target.
Next the ${GemName}.Static
STATIC library target should change to an OBJECT target be renamed to a private ${GemName}.Private.Object
as part of the Gem template(this also applies to the ${GemName}.Editor.Static
being renamed to ${GemName}.Editor.Private.Object
).
Why switch the STATIC library over to a OBJECT target?
An OBJECT target in CMake allow developers to compile source files into a collection of object files without creating a static or shared library. More information on an OBJECT target can be found here.
This has the major benefit that unnecessary copies of object files aren't stored in a static library, thereby saving disk space.
Furthermore this de-clutters the SDK lib
directory as it would only contain public static libraries that aren't meant to be used by other targets.
An example where this helps is with the LyShine.Static
library target which should be private to the LyShine Gem. In a build of the source engine, there would be two copies of the object files that get build for it, one on disk and one inside of the "LyShine.Static.a" archive file, but if the target is an OBJECT target, then there would be no "LyShine.Static.a" file.
Because there is no "LyShine.Static.a" file, when creating an SDK layout, there is no copying of that private library to the SDK
What are the advantages of the suggestion?
The gem maintainer, has a framework in which to expose public functionality, while reserving a section of their gem from making changes to the implementation without worrying that they are breaking external users.
This make it easier to implement a version scheme for code changes within a gem. Perhaps API breaking changes are major version bumps, while internal implementation changes are minor changes
Users of newer gems will now have a known target name they can depend on in their libraries to access the functionality of the gem.
What are the disadvantages of the suggestion?
-
The disadvantage is that existing gems will not have the new API targets.
The expectation for users will be that to add a dependency on a gem API, they would only need to a dependency on ${GemName}.<variant>.API
target.
User could be surprised when coming across gems that don't use the convention. This creates a bit of an inconsistency between older gems and newer gems.
-
Using the updated Gem Template would result in 1 to 2 more TARGETS being created out of the box by default.
If using an IDE such as Visual Studio with a Gem created from the Default Gem template, this will result in 8 .vcxproj being loaded instead of 6. This is balanced with the .Static
vcxproj is being split into a smaller .API
and .Private.Object
vcxproj targets which loads faster.
How will this be work within the O3DE project?
This should be seamless. Users would use the same o3de.py create-gem
command to create their gems with the new public API and private object targets.
Are there any alternatives to this suggestion?
There is a feature request with CMake to allow specification of DIRECTORY scope for regular CMake Targets: https://gitlab.kitware.com/cmake/cmake/-/issues/23433
That would allow enforcement through cmake other gems couldn't depend on the private ${GemName}.Static
targets making it a lot easier to make sure an internal library isn't used outside of the Gem's root directory.
What is the strategy for adoption?
The first step is to update the Gem Templates to expose the new .API
INTERFACE targets with the public include directories, compile option and build dependencies.
The next step is to update existing gems with the O3DE engine repo: https://github.com/o3de/o3de/tree/development/Gems with gem variant aliases for ${GemName}.<variant>.API
targets.
Ideally the existing ${GemName}.Static
targets inside of gems that aren't a dependency of another gem are changed into OBJECT targets and renamed ${GemName}.Private.Object
.
Gems with the engine which are depending on other gems ${GemName}.Static
target should be updated to instead depend on the ${GemName}.API
target. The ${GemName}.API
target for the gem that is being depended on can be an alias of ${GemName}.Static
in this case.
Example: New Gem within .API targets
The following is an example of a TestGem created using the new Public/Private API split
# Currently we are in the Code folder: ${CMAKE_CURRENT_LIST_DIR}
# Get the platform specific folder ${pal_dir} for the current folder: ${CMAKE_CURRENT_LIST_DIR}/Platform/${PAL_PLATFORM_NAME}
# Note: o3de_pal_dir will take care of the details for us, as this may be a restricted platform
# in which case it will see if that platform is present here or in the restricted folder.
# i.e. It could here in our gem : Gems/TestGem/Code/Platform/<platorm_name> or
# <restricted_folder>/<platform_name>/Gems/TestGem/Code
o3de_pal_dir(pal_dir ${CMAKE_CURRENT_LIST_DIR}/Platform/${PAL_PLATFORM_NAME} "${gem_restricted_path}" "${gem_path}" "${gem_parent_relative_path}")
# Now that we have the platform abstraction layer (PAL) folder for this folder, thats where we will find the
# traits for this platform. Traits for a platform are defines for things like whether or not something in this gem
# is supported by this platform.
include(${pal_dir}/PAL_${PAL_PLATFORM_NAME_LOWERCASE}.cmake)
# The TestGem.API target declares the common interface that users of this gem should depend on in their targets
ly_add_target(
NAME TestGem.API INTERFACE
NAMESPACE Gem
FILES_CMAKE
testgem_api_files.cmake
${pal_dir}/testgem_api_files.cmake
INCLUDE_DIRECTORIES
INTERFACE
Include
BUILD_DEPENDENCIES
INTERFACE
AZ::AzCore
)
# Add the TestGem.Private.Object target
# Note: We include the common files and the platform specific files which are set in
# 1.testgem_private_files.cmake
# 2.${pal_dir}/testgem_private_files.cmake
ly_add_target(
NAME TestGem.Private.Object OBJECT
NAMESPACE Gem
FILES_CMAKE
testgem_private_files.cmake
${pal_dir}/testgem_private_files.cmake
INCLUDE_DIRECTORIES
PRIVATE
Include
Source
BUILD_DEPENDENCIES
PUBLIC
AZ::AzCore
AZ::AzFramework
)
# Here add TestGem target, it depends on the Private Object library and Public API interface
ly_add_target(
NAME TestGem ${PAL_TRAIT_MONOLITHIC_DRIVEN_MODULE_TYPE}
NAMESPACE Gem
FILES_CMAKE
testgem_shared_files.cmake
${pal_dir}/testgem_shared_files.cmake
INCLUDE_DIRECTORIES
PUBLIC
Include
PRIVATE
Source
BUILD_DEPENDENCIES
PUBLIC
Gem::TestGem.API
PRIVATE
Gem::TestGem.Private.Object
)
# By default, we will specify that the above target TestGem would be used by
# Client and Server type targets when this gem is enabled. If you don't want it
# active in Clients or Servers by default, delete one of both of the following lines:
ly_create_alias(NAME TestGem.Clients NAMESPACE Gem TARGETS Gem::TestGem)
ly_create_alias(NAME TestGem.Servers NAMESPACE Gem TARGETS Gem::TestGem)
# For the Client and Server variants of TestGem Gem, an alias to the TestGem.API target will be made
ly_create_alias(NAME TestGem.Clients.API NAMESPACE Gem TARGETS Gem::TestGem.API)
ly_create_alias(NAME TestGem.Servers.API NAMESPACE Gem TARGETS Gem::TestGem.API)
# If we are on a host platform, we want to add the host tools targets like the TestGem.Editor MODULE target
if(PAL_TRAIT_BUILD_HOST_TOOLS)
# The TestGem.Editor.API target can be used by other gems that want to interact with the TestGem.Editor module
ly_add_target(
NAME TestGem.Editor.API INTERFACE
NAMESPACE Gem
FILES_CMAKE
testgem_editor_api_files.cmake
${pal_dir}/testgem_editor_api_files.cmake
INCLUDE_DIRECTORIES
INTERFACE
Include
BUILD_DEPENDENCIES
INTERFACE
AZ::AzToolsFramework
)
# The TestGem.Editor.Private.Object target is an internal target
# which is only to be used by this Gem's CMakeLists.txt and any Subdirectories
# Other Gems should not use this target
ly_add_target(
NAME TestGem.Editor.Private.Object OBJECT
NAMESPACE Gem
FILES_CMAKE
testgem_editor_private_files.cmake
INCLUDE_DIRECTORIES
PRIVATE
Include
Source
BUILD_DEPENDENCIES
PUBLIC
AZ::AzToolsFramework
$<TARGET_OBJECTS:Gem::TestGem.Private.Object>
)
ly_add_target(
NAME TestGem.Editor GEM_MODULE
NAMESPACE Gem
AUTOMOC
FILES_CMAKE
testgem_editor_shared_files.cmake
INCLUDE_DIRECTORIES
PRIVATE
Source
PUBLIC
Include
BUILD_DEPENDENCIES
PUBLIC
Gem::TestGem.Editor.API
PRIVATE
Gem::TestGem.Editor.Private.Object
)
# By default, we will specify that the above target TestGem would be used by
# Tool and Builder type targets when this gem is enabled. If you don't want it
# active in Tools or Builders by default, delete one of both of the following lines:
ly_create_alias(NAME TestGem.Tools NAMESPACE Gem TARGETS Gem::TestGem.Editor)
ly_create_alias(NAME TestGem.Builders NAMESPACE Gem TARGETS Gem::TestGem.Editor)
# For the Tools and Builders variants of TestGem Gem, an alias to the TestGem.Editor API target will be made
ly_create_alias(NAME TestGem.Tools.API NAMESPACE Gem TARGETS Gem::TestGem.Editor.API)
ly_create_alias(NAME TestGem.Builders.API NAMESPACE Gem TARGETS Gem::TestGem.Editor.API)
endif()
################################################################################
# Tests
################################################################################
# See if globally, tests are supported
if(PAL_TRAIT_BUILD_TESTS_SUPPORTED)
# We globally support tests, see if we support tests on this platform for TestGem.Tests
if(PAL_TRAIT_TESTGEM_TEST_SUPPORTED)
# We support TestGem.Tests on this platform, add dependency on the Private Object target
ly_add_target(
NAME TestGem.Tests ${PAL_TRAIT_TEST_TARGET_TYPE}
NAMESPACE Gem
FILES_CMAKE
testgem_tests_files.cmake
INCLUDE_DIRECTORIES
PRIVATE
Tests
Source
BUILD_DEPENDENCIES
PRIVATE
AZ::AzTest
AZ::AzFramework
Gem::TestGem.Private.Object
)
# Add TestGem.Tests to googletest
ly_add_googletest(
NAME Gem::TestGem.Tests
)
endif()
# If we are a host platform we want to add tools test like editor tests here
if(PAL_TRAIT_BUILD_HOST_TOOLS)
# We are a host platform, see if Editor tests are supported on this platform
if(PAL_TRAIT_TESTGEM_EDITOR_TEST_SUPPORTED)
# We support TestGem.Editor.Tests on this platform, add TestGem.Editor.Tests target which depends on
# private TestGem.Editor.Private.Object target
ly_add_target(
NAME TestGem.Editor.Tests ${PAL_TRAIT_TEST_TARGET_TYPE}
NAMESPACE Gem
FILES_CMAKE
testgem_editor_tests_files.cmake
INCLUDE_DIRECTORIES
PRIVATE
Tests
Source
BUILD_DEPENDENCIES
PRIVATE
AZ::AzTest
Gem::TestGem.Private.Object
)
# Add TestGem.Editor.Tests to googletest
ly_add_googletest(
NAME Gem::TestGem.Editor.Tests
)
endif()
endif()
endif()
If using the CMake Tools extension for VSCode, this is how a new Gem Targets would look
![image](https://user-images.githubusercontent.com/56135373/166524094-a045ebf4-0b33-427e-80d4-78261413ab5f.png)
#Example: Existing Gem using an API target
The following is a contrived example of the NvCloth gem using the API target of the TestGem from the example above.
The gem maintainer of the NvCloth gem would add a build dependency on the TestGem.${API target for the runtime API target.
ly_add_target(
NAME NvCloth.API INTERFACE
NAMESPACE Gem
FILES_CMAKE
nvcloth_api_files.cmake
INCLUDE_DIRECTORIES
INTERFACE
Include
BUILD_DEPENDENCIES
INTERFACE
3rdParty::NvCloth
AZ::AzFramework
Gem::AtomLyIntegration_CommonFeatures.API
Gem::TestGem.API
Gem::EMotionFX.API
)
Other Concerns
The O3DE repo currently contains 80 top level gems with an AutomatedTesting project which contains a single gem.
The Atom Gem contains several sub gems(Atom_RHI, Atom_RPI, etc...) as well which pushes the total number of gems to over 100.
The list of targets in gems vary based on the needs of that gem what it wishes to expose. Some gems have 0 targets others have 2~3 targets based on if Testing is enabled, while others might have 9+ targets that will show up in an IDE such as Visual Studio.
Gems are plugins, that will often have disjoint authors, that should not need to worry about any issues with a particular when authoring their gems.
For example a gem author who creates a Particle system gem that has 10 targets and host it as part of an external repo, does not need to worry if their gem can load within the engine's build solution, as only gems registered with the the current project and the engine have their CMakeLists.txt visited.
But someone adding a new gem within the engine repo, will have that gem's targets be part of the generated build solution for whatever generator was used(Ninja, Xcode, Unix Makefiles, etc...).
A subset of gems registered within the engine and resides within the O3DE repo should be moved to another repo, since they are not core to the engine itself.
The following is a list of gems that might not be core to the engine itself can be moved to another repo
Gem List
Achievements
AssetValidation
AtomContent
AtomTressFX
AudioEngineWwise
AWSClientAuth
AWSCore
AWSGameLift
AWSMetrics
BarrierInput
Blast
CertificateManager
CustomAssetExample
DevTextures
ExpressionEvaluation
GameState
GameStateSamples
Gestures
HttpRequestor
InAppPurchases
LocalUser
LyShineExamples
MessagePopup
Metastream
Microphone
MotionMatching
NvCloth
PhysXDebug
Presence
PrimitiveAssets
SaveData
SceneLoggingExample
ScriptCanvasDeveloper
ScriptCanvasPhysics
ScriptCanvasTesting
ScriptedEntityTweener
ScriptEvents
SliceFavorites
Stars
StartingPointCamera
StartingPointInput
StartingPointMovement
TestAssetBuilder
TickBusOrderViewer
Twitch
UiBasics
VideoPlaybackFramework
VirtualGamepad
That would leave the engine repo with set of "core" gems
Atom
AtomLyIntegration
AudioSystem
Camera
CameraFramework
CrashReporting
DebugDraw
EditorPythonBindings
EMotionFX
FastNoise
GradientSignal
GraphCanvas
GraphModel
ImGui
LandscapeCanvas
LmbrCentral
LyShine
Maestro
Multiplayer
PhysX
Prefab
Profiler
PythonAssetBuilder
QtForPython
SceneProcessing
ScriptCanvas
SurfaceData
Terrain
TextureAtlas
Vegetation
WhiteBox
That would reduce the number of gems that come with the engine from 80 to 32.