[Feature] Entity Blueprints about entitas

[trumpets]

I have implemented something similar for behaviour trees in Unity, was working on a project that required that level of AI and while in theory is amazing for combining common behaviours (entities in this example) it turned out to be just a bunch of objects being created and cleaned up by the GC ( we were creating a lot of them ). IMHO I am pretty much satisfied by just writing pool extensions which accomplish the same thing. Instead of a class and an object I just have an extension method :)

[npruehs]

The blueprints are just created once, when the game is loaded, and never cleaned up again :) The idea is as follows:

1. Game designer creates a blueprint, say "Knight".
2. Game designer adds required components to the blueprint, say "HealthComponent" and "AttackComponent".
3. Game designer tweaks the values, say settings Health to 140.
4. Game designer saves the blueprints to disk.
5. Entitas loads all blueprints at startup.
6. Whenever the game needs to create a new Knight, it looks up the respective blueprint, and creates a new entity.

This behaviour is similar to the CreateRandomPiece extension method in the MatchOne example, except for all values are easily configured and re-balanced on a hour-to-horr basis by the game designers. We found that approach to be very important (or even required) when building games that require a lot of iteration, such as real-time strategy games.

[MrMonotone]

@npruehs How are blueprints different to extensions? Extensions can be created without creating a new derived type, recompiling, or otherwise modifying the original type. I mean if you want to create a new system class that generates that object(factory). See this for reference: https://github.com/sschmid/Entitas-CSharp-Example/blob/master/Assets/Sources/Features/CreateOpponentsSystem.cs

Maybe I am missing something?

[npruehs]

@MrMonotone Creating a new extension is not a big deal in most cases for us engineers. They require a re-compile however, which can take longer the bigger your project gets (~ 1 minute in our current project).

For game designers, that are purely concerned with adding new data and balancing existing one, this can be a huge deal, though. First, we don't want designers to mess around in our C# code. Second, many designer just won't have Visual Studio or a similar IDE installed on their system. Imagine your teammate wanting to tweak the Health and Damage values of some of your entities. It would be far easier with an editor similar to, say, StarCraft II, than doing so in code:

Blueprints allow this entity data to be easily serialized and de-serialized, from pure data files (e.g. JSON). This reduces the turn-around time by a lot, as we have learned.

[sschmid]

Hey, I thought about it and I think this could be very useful in some scenarios. Do you already have a system to convert back and fourth or shall I provide sth?

[npruehs]

We've already got something in place for converting between XML and C# blueprints.

Then, I'd love to provide the conversion between blueprints and Entitas entities. My original question was whether my proposed way would be the way to go, and if so, whether I should wait for the new code generator to be finished.

Also, if you have a different data format in mind, we could split the work :)

[sschmid]

Your described suggestion def makes sense to me.
I don't think the code generator is affected by this feature, or am I missing sth.

I was thinking to create a new subfolder in the repo called "Add-ons" where tools like BluePrints can be collected. There you can provide a new feature with tests and readme.

I will add an example soon.

[npruehs]

The challenge is creating the components (step 2.iii.). Because their types are not known at compile-time, we were forced to use reflection in our framework to create the entities. While this works well, a common interface and a method generated with the code generator would be much faster. But I can start off either way, and we can iterate on it.

I love the idea with the feature sub-folder. We could even go for an additional repository - that depends on how invasive that feature feels for you.

Looking forward to the example!

[MrMonotone]

@sschmid @npruehs have you two started working on this?

[npruehs]

I could, anytime basically - thought I'd wait for the Add-ons example, before messing up with your repo structure.

[sschmid]

Hi Nick, I was thinking about how provide a nice and streamlined way for people to contribute cool add-ons like BluePrints. I couldn't find a nice solution that is super easy to maintain yet. I was thinking to add a Add-ons folder in the repo, or creating a new repo for Add-ons etc. But I think it'll be difficult to maintain when I release new versions. Tools might get out of date if they don't get updated. We could quickly end up with different tools relying on different versions of Entitas. I could migrate the tools when I update Entitas, but that doesn't scale well :) Maybe you have some input how we could solve this. The easiest solution for now could be gathering links to all the contributed tools in the wiki or in a Add-ons.md. So everyone would work on their tools in their own repo. On the other hand I still think it would be cool to have a central place for that... What do you think?

[MrMonotone]

@sschmid I do not think there is anyway to solve software not getting updated. There is always going to be a chance that old addons don't get updated because there is no way to predict if how the API will change. Unless you are very firm with keeping backwards compatibility but its probably too early to worry about that. The way you could mitigate the damage is just to have Entitas not recognize old versions of addons.
What I would do:
There is a way to embed "subrepos" in git. I forgot what they were called.
What I would do is have addons be a subrepo of Entitias. Addons would contain subrepos to various addon repos. Then authors would send pull requests updating those subrepos by commit. Then you would just accept those. Entitas would only recognize addons that are compatible with the current version of Entitas being used. Maybe give some sort of warning like this project is incompatible or something. You could do a similar thing like you have with the update button in Unity haha.
I am only a uni student so take what I say with a grain of salt.

[npruehs]

Hey @sschmid :) I think the cleanest way to go would be to have separate repositories. As @MrMonotone pointed out, compatibility will always be an issue - can't escape that. Luckily, that's not your problem. Plugin developers should always state the version of Entitas they are building against, and are responsible of updating whenever Entitas introduces breaking changes. See https://github.com/joscha/play-authenticate for an example, this is an authentication plugin for the Play! framework we've been using a few years ago.

Having a list in the wiki or Addons.md would be great for both sides, I think 👍

I don't think that including add-ons in Entitas itself via git submodules or subtrees is a good idea, though. People should be able to choose which plugins they want to use, and not be forced to include all of them when cloning Entitas. In most cases, if the plugin developers are working cleanly, Entitas won't even need to have to "recognize" or "accept" these add-ons - they would just build on Entitas classes or, even better, used in addition to them. If Entitas adheres to common Framework Design Guidelines (as it seems to me :) ), then this should not be much of an issue. (See https://msdn.microsoft.com/en-us/library/ms229042%28v=vs.110%29.aspx for some pointers.)

Whenever something really gets popular and no one wants to miss it, you can still integrate it into Entitas itself. I've seen that happening for git subtrees recently: https://github.com/apenwarr/git-subtree/blob/master/THIS-REPO-IS-OBSOLETE

[chrischu]

I think it is a good idea to provide a way for people to add-on to Entitas functionality, however I also think that those add-ons don't need to live in the same git repository. In my opinion and approach similar to the plug-ins folder of Notepad++ (or other countless software that offers plug-in support) would be best. This way people can write add-ons and offer them to the community to use (Entitas can help them promote the add-ons by linking to them in the wiki) but ultimately the Entitas USER gets to decide which add-ons he needs (and therefore copies into the add-ons folder in his Entitas "installation"). This way Entitas is not reliable for making sure all the add-ons continue to function with every version.

However, if you decide to actually go for the add-on approach that would make them part of your repository, I would not go for git submodules. They have very little (if any) advantages over the tried and tested feature-branch&pull-request approach, and might even less good. For example I'm not sure for example if github would show pull-requests just containing submodule updates correctly (show all the changes in the submodule etc.).

[iaincarsberg]

Wouldn't using semver's major number solve the API change issues?

As a way to automate the flagging of obsolete code, you could force all addon's to provide something akin to npm's package.json, which lists supported versions of Entitas, using strings such as ">0.10.0&<=0.27.0", this would also help when you have add-ons depending on other add-ons.

Then the migration tool could report errors when migrating to a version of Entitas that isn't compatible with older versions of installed add-ons.

[MrMonotone]

If we are going to seperate "Addons" from Entitas then we need to decide what specifies a feature of Entitas or what is an Addon. If Entitas is suppose to be more data driven then wouldn't Blueprints be considered a feature instead of an Addon? For example is logging a feature or and addon? There could be arguments for both. Where is the line drawn? I do not think its a big issue now but I feel it will be a tough issue to answer down the line.

[npruehs]

I think things like blueprints can easily built on top of Entitas, without having to change its core. We did the same thing in our framework, and made the mistake at the beginning of coupling it too closely to the core where it wasn't necessary at all. Clearly, blueprints don't do anything special except for parsing data from any stream (could be network, file, ...) and calling Entitas core functionality for creating entities. Other addons should be designed the same way, in order to minimize the dependencies of the Entitas core, avoiding it to have to reference stuff like IO or Reflection.

[sschmid]

Yes, I can imagine BluePrints being an Add-on. Add-ons should work with the current version of Entitas without having to change the core. Logging is a good example which can be an Add-on but also might require some changes in Entitas (e.g. adding more event hooks). In this case I would suggest to create a new issue or PR to discuss changes.

I suggest we start easy first, meaning you can create your own repo and I can create a Add-ons.md where I can link to it. Then we can see how it goes. If needed, we can always think about adding nice tooling to discover and update installed add-ons, some sort of package manager. But let's start easy first :)

[sschmid]

@npruehs Let me know when you create a repo for BluePrints. I guess you already have a lot of concrete ideas about it, I'd love to hear them. I have some ideas, too.
I'm currently also working on the code generator to decouple the generation logic by adding an intermediate data structure. I'll keep you updated.

[sschmid]

See #62

[npruehs]

Yeah, the ideas are at the top of this thread :) I'll keep you posted!

[npruehs]

Alright, it's almost done! I've created a new repository at https://github.com/npruehs/Entitas-Blueprints-CSharp - are you alright with the name? Everything, including the Readme file still work in progress of course.

I've been diving right into your code generation and I have to admit its interface works like a charm. I was able to write my own code generator, and hook it up to the Match-One example at https://github.com/sschmid/Match-One

I love that approach because it's not invasive in any kind, meaning that I didn't need to change any Entitas classes at all.

Here's a gist of the results, works incredibly well already:

https://gist.github.com/npruehs/c3270996b178868dbaa1

Note that the Match-One example seems to be still missing the methods RemoveComponentSuffix and ComponentLookupTags, I had to copy them over from the latest Entitas release.

During development, I've opened a few issues, two of which might be helpful to reduce code duplication:

#64
#65 - would improve Entitas.Blueprints
#66 - would improve Entitas.Blueprints

I'm gonna clean up the code now, rename the output files and so on, and I'm looking forward to hearing your feedback.

Good night everyone ;)

[SvDvorak]

Nice, looks pretty solid! There is one thing though that wouldn't work in my case; it looks like the current code requires each property to have a unique name. For example, if the PositionComponent has x & y then no other component can have fields with those names. Could be fixed with a composite key of type and property-name.

[npruehs]

Oh, yeah. My bad - in our framework implementation we do it exactly that way. By restrictions of the C# language, this guarantees all keys to be unique. I'd have stumbled upon this as soon as I'd started the implementation of the serialization part ;)

[SvDvorak]

No biggie. And yeah, you probably would have noticed when doing serialization. Thought I'd sound all smart and point it out beforehand =)

[npruehs]

Yeah thank you! If we hadn't had the same issue before in our projects, maybe I wouldn't have come up with a solution as nice as the one you've proposed 👍

[npruehs]

Alright, XML serialization is done, and so is support for singleton components (components without any fields).

I've updated the gist, take a look at the result data file:

https://gist.github.com/npruehs/c3270996b178868dbaa1

@sschmid Any chance that some mapping similar to ComponentNameToId will be added to the core of Entitas? Should I create an issue for that? Or should I keep it locally?

Next, I'm going to create a version of Match-One that uses blueprints instead of PoolExtensions class for creating pieces, and I'm going to clean up my code as soon as the new code generator of Entitas is ready :)

[ForgeMistress]

I found a system like this actually rather easy to make using C#'s built-in serialization, though an official version of the concept would be incredibly useful.

In my honest opinion, this is pretty much the only feature that is missing from Entitas to make it viable for large-scale projects.

[sschmid]

I can also see that becoming a part of Entitas :)

[trumpets]

To be honest, as time goes by our game needs this more and more :)

[Backman]

After reading this issue I could not stop thinking about it. Therefore, a couple of weeks ago, I started to implement a similar system which I was thinking to use in my side project.

Did not come here to hijack the comment feed but thought it might be a good idea to share it with you guys.
It's probably not as robust as your implementation, and very early in development, but it might give you some ideas.
Great idea to use the exisiting code generation interface. That idea did not cross me.

Here's a link to the repository: https://github.com/Backman/entitas-blueprint
At the moment the editor is kinda broken but will try to fix it as soon as possible.

[npruehs]

@Backman Sure, will do! Any merits of writing your own BlueprintManager and FieldTable instead of just using plain old Dictionaries? We did the same for our project, but I can't seem to find a real advantage of it.

@sschmid Any updates on my previous question? :)

[Backman]

@npruehs No, there really is no reason I use classes instead of Dictionaries. Just kinda made sense to do it when I was writing it! :)

[sschmid]

@npruehs Having ComponentNameToId built in makes sense. I can add it

[sschmid]

Btw, I start playing around integrating Blueprints... :)

[sschmid]

For me it's important that all features of Entitas also work without the code generator. I'd like to keep it optional (although I cannot image to work without it anymore :)) I try to find a solution that works also without the code generator

[dxslly]

This is a great feature idea. A thought I have not seen mentioned yet is parenting blueprints. This would allow blueprints to inherit the collection of components and values of the parent blueprint as defaults. Child blueprints could override and add these components. This would allow specializations of blueprints without having to duplicate inherited components and help avoid large refactoring of similar blueprints.

It maybe worth keeping in mind another possible feature while designing this due to how similar they may be. That feature being serializing and deserializing of entity instances. I could see this being used for persisting game state as save files or into a database.

[npruehs]

@sschmid Yeah, the single approach we've been coming up with in order to avoid code generation was to use reflections for creating the components. The remaining parts mostly stay the same.

@dxslly Yes, we've got both of that as well. First one is great for reducing redundancy, but introduces the same drawbacks as classical inheriance-based game models. In a recent game, we came up with Fighter blueprints, and Building blueprints, and very soon found out that we need a Turret blueprint which would have to derive from both ;) But because it's still useful more often than not, we can go for it.

The second one we've been using for level editors as well. We called it an Entity Configuration, which essentially boils down to an Entity Blueprint + an additional dictionary with overriding values. That way, you can put a "Wounded Knight" in your level, which is using the same values as a Knight but has reduced life at the beginning.

[npruehs]

@sschmid Just updating to Entitas 0.28.2. Already love the new ComponentInfo provided to code generators! Any way of accessing whether the component has Entitas.CodeGenerator.DontGenerateAttribute attached? You don't seem to check it when creating ComponentInfos, and ComponentInfo doesn't provide a reference to the underlying component type, or am I missing something? :)

[sschmid]

see https://github.com/sschmid/Entitas-CSharp/blob/master/Entitas.CodeGenerator/Entitas.CodeGenerator/Intermediate/ComponentInfo.cs#L9-L10

[sschmid]

Thanks! :) Yes ComponentInfo will make our lives easier I hope. It will make the switch to the new generator very easy

[sschmid]

string fullTypeName is used instead of System.Type

[sschmid]

This is checking for DontGenerate
https://github.com/sschmid/Entitas-CSharp/blob/master/Entitas.CodeGenerator/Entitas.CodeGenerator/Providers/TypeReflectionProvider.cs#L33-L34

[sschmid]

It's string based now so I can generate from different assemblies

[npruehs]

I see! Nice. What is generateIndex for, in contrast to generateMethods?

[sschmid]

generateMethods determines whether the usual extensions get generated.
generateIndex determines whether an entry to ComponentIds gets added.

if for some reason you don't want to have methods generated you still might need the index though

[npruehs]

Thanks! Might wanna add that to the property XML comments :)

[npruehs]

@sschmid There you go! The initial release: https://github.com/npruehs/Entitas-Blueprints-CSharp

Now for a few more questions:

• Right now, the code generators all write their files to the same destination folder. It might come in handy in the future to allow generators to write files to a subfolder of "Generated". Maybe add a third property to CodeGenFile?
• In the blueprints code generator, I'm adding an additional file header containing the <auto-generated> tag to have code style tools ignore these files (see updated gist: https://gist.github.com/npruehs/c3270996b178868dbaa1). Might wanna do that as well in the base Entitas code generators :)
• I'd love to create a (very similar) version of Match-One with blueprints. Just providing a few blueprints for different pieces, maybe with different colors tied to different scores or something like that. Match-One would then pick a random blueprint instead of just a random asset when creating new pieces. How should we do that? Should I make an additional repository for that as well?
• Finally, I'd like to add some documentation about how to use the blueprints. Should we try and find a way to use them without code generation? Maybe in a different thread, before this one becomes too crowded?

[sschmid]

@npruehs Cool!

• A code generator can set a fixed subfolder by prefixing the CodeGenFile.filename, e.g. "Blueprints/"
• cool, I'll add <auto-generated>
• A Blueprints Match-One Demo would be aweseome to see Blueprints in action :) Own repo sounds good
• Definitely, see #67 where I sketched something already to get a feel for it

[sschmid]

Added <auto-generated>
see 9c8ca63

[npruehs]

Alright, there you go:

https://github.com/npruehs/Match-One-Blueprints

Works like a charm! :) Pieces are read by a new BlueprintsController from the file Assets/Data/blueprints.xml.

Unfortunately, prefixing the generated source file name with a folder name does not work:

DirectoryNotFoundException: Could not find a part of the path "D:\Dev\SCM\Match-One\Assets\Sources\Generated\Blueprints\PositionComponentGeneratedBlueprintsExtension.cs".
System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.IO/FileStream.cs:292)
System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share)
(wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
System.IO.StreamWriter..ctor (System.String path, Boolean append, System.Text.Encoding encoding, Int32 bufferSize) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.IO/StreamWriter.cs:124)
System.IO.StreamWriter..ctor (System.String path, Boolean append, System.Text.Encoding encoding)
(wrapper remoting-invoke-with-check) System.IO.StreamWriter:.ctor (string,bool,System.Text.Encoding)
System.IO.File.WriteAllText (System.String path, System.String contents, System.Text.Encoding encoding) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.IO/File.cs:626)
System.IO.File.WriteAllText (System.String path, System.String contents) (at /Users/builduser/buildslave/mono-runtime-and-classlibs/build/mcs/class/corlib/System.IO/File.cs:621)
Entitas.CodeGenerator.CodeGenerator.writeFiles (System.String directory, Entitas.CodeGenerator.CodeGenFile[] files) (at Assets/Libraries/Entitas/Entitas.CodeGenerator/Editor/CodeGenerator.cs:56)
Entitas.CodeGenerator.CodeGenerator.Generate (ICodeGeneratorDataProvider provider, System.String directory, Entitas.CodeGenerator.ICodeGenerator[] codeGenerators) (at Assets/Libraries/Entitas/Entitas.CodeGenerator/Editor/CodeGenerator.cs:19)
Entitas.CodeGenerator.TypeReflection.TypeReflectionCodeGenerator.Generate (System.Reflection.Assembly assembly, System.String[] poolNames, System.String directory, Entitas.CodeGenerator.ICodeGenerator[] codeGenerators) (at Assets/Libraries/Entitas/Entitas.CodeGenerator.TypeReflection/Editor/TypeReflectionCodeGenerator.cs:7)
Entitas.Unity.CodeGenerator.UnityCodeGenerator.Generate () (at Assets/Libraries/Entitas/Entitas.Unity.CodeGenerator/Editor/UnityCodeGenerator.cs:27)

Also, Match-One asset serialization is still set to "Mixed", you might want to change it to "Force Text" for future updates :)

I've found one or two other things as well, but I think I'm gonna create a few new issues, so you can finally close this one.

[sschmid]

@npruehs how did you actually create the blueprints.xml?
Does it support custom classes?

[npruehs]

@sschmid The XML file has been created with the Entitas Blueprints serializer:

https://github.com/npruehs/Entitas-Blueprints-CSharp/blob/develop/Source/Entitas.Blueprints/Entitas.Blueprints.Xml/BlueprintXmlSerializer.cs

That serializer essentially wraps the .NET XmlSerializer and thus supports all CLR classes, with the same restrictions applied (i.e. serializes public fields and properties only, requires getter and setter, ...)

Of course, as soon as one or more official Entitas formats have been developed, I'd love to add them to the Tome editor for easier data editing :)

[sschmid]

I started playing around with SerializedObject and using the EntityDrawer known from VisualDebugging which basically already provides most of the features to create a Blueprint out of the box. The first steps look promising and I try to also support GameObjects and other Unity Objects.

[sschmid]

Maybe we can even use Unity’s serialization system with its yaml backend

[npruehs]

Nice! One thing though: Many games I've been working at recently required the data (or parts of it) to be present at the server as well, which obviously is unable to deal with ScriptableObjects. I'd suggest providing one non-Unity data format at least to keep Entitas interesting for online games, and/or other engines such as MonoGame.

EDIT: Haha, but I love the animation curve component! :)

[sschmid]

@npruehs Definitely agree! It was just so tempting to play around with it :)

[sschmid]

Updated Wiki with a link to your repo
https://github.com/sschmid/Entitas-CSharp/wiki/Tools-and-Extensions