hexdecimal / python-tcod-ecs

Python sparse-set ECS with strong type-hinting. Supports entitiy relations.

About

tcod-ecs is a Sparse-set Entity-component-system implemented using Python's dict and set types. See the ECS FAQ for more info.

This implementation focuses on type-hinting, organization, and is designed to work well with Python. The following features are currently implemented:

• Entities can store components which are instances of any Python object. Components are looked up by their type.
• Entities can have one instance of a type, or multiple instances of a type using a hashable tag to differentiate them.
• Entity relationships are supported, either as many-to-many or many-to-one relationships.
• ECS Queries can be made to fetch entities having a combination of components/tags/relations or excluding such.
• The ECS Registry object can be serialized with Python's pickle module for easy storage.

A lightweight version which implements only the entity-component framework exists called tcod-ec. tcod-ec was geared towards a dynamic-typed-dict style of syntax and is missing a lot of important features such as queries and named components.

Installation

Use pip to install this library:

pip install tcod-ecs

If tcod is installed and the version is less than 14.0.0 then import tcod.ecs will fail. Remove or update tcod to fix this issue.

Examples

Registry

The ECS Registry is used to create and store entities and their components.

>>> import tcod.ecs
>>> registry = tcod.ecs.Registry()  # New empty registry

Entity

Each Entity is identified by its unique id (uid) which can be any hashable object combined with the registry it belongs. New unique entities can be created with Registry.new_entity which uses a new object() as the uid, this guarantees uniqueness which is not always desireable. An entity always knows about its assigned registry, which can be accessed with the Entity.registry property from any Entity instance. Registries only know about their entities once the entity is assigned a name, component, tag, or relation.

>>> entity = registry.new_entity()  # Creates a unique entity using `object()` as the uid
>>> entity
<Entity(uid=object at ...)>
>>> entity.registry is registry  # Registry can always be accessed from their entity
True
>>> registry[entity.uid] is entity  # Entities with the same registry/uid are compared using `is`
True

# Reference an entity with the given uid, can be any hashable object:
>>> entity = registry["MyEntity"]
>>> entity
<Entity(uid='MyEntity')>
>>> registry["MyEntity"] is entity  # Matching entities ALWAYS share a single identity
True

Use Registry.new_entity to create unique entities and use Registry[x] to reference a global entity or relation with an id. registry[None] is recommend for use as a global entity when you want to store components in the registry itself.

Do not save the uid's of entities to be used later with registry[uid], this process is slower than holding onto the Entity instance.

Serialization

Registries are normal Python objects and can be pickled as long as all stored components are pickleable.

>>> import pickle
>>> pickled_data: bytes = pickle.dumps(registry)
>>> registry = pickle.loads(pickled_data)

Stability is a priority but changes may still break older saves. Backwards compatibility is not a priority, pickled registries should not be unpickled with an older version of the library. This project follows Semantic Versioning, major version increments will break the API, the save format or both, minor version increments may break backwards compatibility. Check the changelog to be aware of format changes and breaks. There should always be a transition period before a format break, so keeping up with the latest version is a good idea.

Components

Components are instances of any Python type. These can be accessed, assigned, or removed from entities via the dict-like Entity.components attribute. The type is used as the key to access the component. The types used can be custom classes or standard Python types.

>>> import attrs
>>> entity = registry.new_entity()
>>> entity.components[int] = 42
>>> entity.components[int]
42
>>> int in entity.components
True
>>> del entity.components[int]
>>> entity.components[int]  # Missing keys raise KeyError
Traceback (most recent call last):
  ...
KeyError: <class 'int'>
>>> entity.components.get(int, "default")  # Test keys with `.get()` like a dictionary
'default'
>>> @attrs.define
... class Vector2:
...     x: int = 0
...     y: int = 0
>>> entity.components[Vector2] = Vector2(1, 2)
>>> entity.components[Vector2]
Vector2(x=1, y=2)
>>> entity.components |= {int: 11, Vector2: Vector2(0, 0)}  # Multiple values can be assigned like a dict
>>> entity.components[int]
11
>>> entity.components[Vector2]
Vector2(x=0, y=0)

# Queries can be made on all entities of a registry with matching components
>>> for e in registry.Q.all_of(components=[Vector2]):
...     e.components[Vector2].x += 10
>>> entity.components[Vector2]
Vector2(x=10, y=0)

# You can match components and iterate over them at the same time.  This can be combined with the above
>>> for pos, i in registry.Q[Vector2, int]:
...     print((pos, i))
(Vector2(x=10, y=0), 11)

# You can include `Entity` to iterate over entities with their components
# This always iterates over the entity itself instead of an Entity component
>>> for e, pos, i in registry.Q[tcod.ecs.Entity, Vector2, int]:
...     print((e, pos, i))
(<Entity...>, Vector2(x=10, y=0), 11)

Named Components

Only one component can be assigned unless that component is given a unique name. You can name components with the key syntax (name, type) when assigning components. Names are not limited to strings, they are a tag equivalent and can be any hashable or frozen object. The syntax [type] and [(name, type)] can be used interchangeably in all places accepting a component key. Queries on components access named components with the same syntax and must use names explicitly.

>>> entity = registry.new_entity()
>>> entity.components[Vector2] = Vector2(0, 0)
>>> entity.components[("velocity", Vector2)] = Vector2(1, 1)
>>> entity.components[("velocity", Vector2)]
Vector2(x=1, y=1)
>>> @attrs.define(frozen=True)
... class Slot:
...     index: int
>>> entity.components |= {  # Like a dict Entity.components can use |= to update items in-place
...     ("hp", int): 10,
...     ("max_hp", int): 12,
...     ("atk", int): 1,
...     str: "foo",
...     (Slot(1), str): "empty",
... }
>>> entity.components[("hp", int)]
10
>>> entity.components[str]
'foo'
>>> entity.components[(Slot(1), str)]
'empty'

# Queries can be made on all named components with the same syntax as normal ones
>>> for e in registry.Q.all_of(components=[("hp", int), ("max_hp", int)]):
...     e.components[("hp", int)] = e.components[("max_hp", int)]
>>> entity.components[("hp", int)]
12
>>> for e, pos, delta in registry.Q[tcod.ecs.Entity, Vector2, ("velocity", Vector2)]:
...     e.components[Vector2] = Vector2(pos.x + delta.x, pos.y + delta.y)
>>> entity.components[Vector2]
Vector2(x=1, y=1)

Tags

Tags are hashable objects stored in the set-like Entity.tags. These are useful as flags or to group entities together.

>>> entity = registry.new_entity()
>>> entity.tags.add("player")  # Works well for groups
>>> "player" in entity.tags
True
>>> entity.tags.add(("eats", "fruit"))
>>> entity.tags.add(("eats", "meat"))
>>> set(registry.Q.all_of(tags=["player"])) == {entity}
True

Relations

Use Entity.relation_components[component_key][target] = component to associate a target entity with a component. Use Entity.relation_tag[tag] = target to associate a tag exclusively with a target entity. Use Entity.relation_tags_many[tag].add(target) to associate a tag with multiple targets.

Relation queries are a little more complex than other queries. Relation tags and relation components share the same space then queried, so 'normal' tags should not be in the format of a component key. Relations are unidirectional, but you can query either end of a relation.

>>> @attrs.define
... class OrbitOf:  # OrbitOf component
...     dist: int
>>> LandedOn = "LandedOn"  # LandedOn tag
>>> star = registry.new_entity()
>>> planet = registry.new_entity()
>>> moon = registry.new_entity()
>>> ship = registry.new_entity()
>>> player = registry.new_entity()
>>> moon_rock = registry.new_entity()
>>> planet.relation_components[OrbitOf][star] = OrbitOf(dist=1000)
>>> moon.relation_components[OrbitOf][planet] = OrbitOf(dist=10)
>>> ship.relation_tag[LandedOn] = moon
>>> moon_rock.relation_tag[LandedOn] = moon
>>> player.relation_tag[LandedOn] = moon_rock
>>> set(registry.Q.all_of(relations=[(OrbitOf, planet)])) == {moon}
True
>>> set(registry.Q.all_of(relations=[(OrbitOf, ...)])) == {planet, moon}  # Get objects in an orbit
True
>>> set(registry.Q.all_of(relations=[(..., OrbitOf, None)])) == {star, planet}  # Get objects being orbited
True
>>> set(registry.Q.all_of(relations=[(LandedOn, ...)])) == {ship, moon_rock, player}
True
>>> set(registry.Q.all_of(relations=[(LandedOn, ...)]).none_of(relations=[(LandedOn, moon)])) == {player}
True

Relation queries

You can use the following table to help with constructing relation queries. tag is a component key if you are querying for a component relation.

Includes	Syntax
Entities with a relation tag to the given target	(tag, target_entity)
Entities with a relation tag to any target	(tag, ...) (Literal dot-dot-dot)
Entities with a relation tag to the targets in the given query	(tag, registry.Q.all_of(...))
The target entities of a relation of a given entity	(origin_entity, tag, None)
The target entities of any entity with the given relation tag	(..., tag, None) (Literal dot-dot-dot)
The target entities of the queried entities with the given relation	(tag, registry.Q.all_of(...))

python-tcod-ecs's Issues

Callbacks on state changes

There should be support for callbacks on specific state changes such as a component being added or removed:

@attrs.define(frozen=True)  # Class is frozen to ensure that a changed value is announced
class Position:
    x: int
    y: int


@tcod.ecs.register_on_component_changed(Position)
def on_changed_position(entity: Entity, old_pos: Position | None, new_pos: Position | None) -> None:
    """Check for Position changes in all worlds."""
    if old_pos is not None and new_pos is not None:
        print(f"Changed from {old_pos} to {new_pos}")
    elif old_pos is None and new_pos is not None:
        print(f"{new_pos} added")
    else:
        assert old_pos is not None and new_pos is None
        print(f"{old_pos} removed")

This is global, there should also be a per-world variant.

These types of callbacks can also be used to manage compiled queries in #3

Would also need to add callbacks for tags and relations.

These should probably not be called during unpickling.

• On component changes
• On tag added/removed
• On relation added/removed
• On relation component changes
• World-specific callbacks
• Entity added/removed to compiled query

A Roadmap to Entity Relationships

I've been using Sander Mertens articles to implement relationships. I've implemented these in the basic sense but they also have an article that describes a roadmap for a more complete implementation.

• Components as entities - Python objects natively support what this is trying to do
• #6
• Support relationship pairs in archetype storage
• Relationship components - Already added
• Wildcard queries - Half of this is implemented
• The component index
• Cleanup
• Cleanup traits
• Multi-source queries
• Relationship traversal
• Query cache revalidation
• Breadth first traversal
• Uncached queries
• Multi component observers
• Event propagation
• Empty table optimization
• Garbage collection
• Rule engine
• Exclusive relationships - Already added as an alternative syntax for relationship access
• #12
• Query DSL - Probably not needed for a Python library

I don't know if I'll ever complete this. Some of these might be too slow when implemented in pure-Python and if I were to make a C extension then I may as well just port Flecs itself to Python.

It's clear that these features will be incredibly useful once added. At the very least I should keep all of these in mind as I work on this module.

`cattrs` support

I was asked to support cattrs and other generic serialization libraries. With the boilerplate required to do this externally it would be better to add cattrs support directly to this package instead.

This has to do the following:

• Save and load anonymous Entity objects which are always a (world, uid) pair.
• Save and load anonymous objects without knowing their type ahead of time. These may be components, tags, or uid's.

Code examples from @slavfox

How to handle object() as uid?

# EntityId: some opaque type with a zero-argument constructor EntityId()
# ecs.entdict: dict[EntityId, collection[Any]]

# note: only typechecks with --enable-recursive-aliases
JsonValue: TypeAlias = dict[str, "JsonValue"] | list["JsonValue"] | int | float | bool | str | None

def serialize_ecs(ecs):
    idmap: dict[EntityId, int] = {id_: i for i, id_ in enumerate(ecs.entdict)}
    return {
        idmap[id_]: [serialize_value(c, idmap=idmap) for c in components] 
        for id_, c in ecs.entdict.items()
    }

def serialize_value(value: Any, idmap: dict[EntityId, int]) -> JsonValue:
    match value:
        case int(_) | float(_) | bool(_) | str(_) | None:
            return value
        case [*xs]:
            return [serialize_value(x, idmap) for x in xs]
        case {**xs}:
            return {str(k): serialize_value(v, idmap) for (k, v) in xs}
        case EntityId():
            return idmap[value]
        case _:
            return {
                field.name: serialize_value(field.value) for field in get_ecs_fields(value)
            }

How to handle component types held by World?

The common pattern for using cattr is tagged enums, which you can think of as, roughly:

serializers = {}
deserializers = {}

# Decorator to register a class with cattr
def cattr(cls_):
    serializers[cls_] = ...
    cls_key = make_cls_key(cls_)
    deserializers[cls_key] = ...
    ...
    return cls_


def serialize(inst):
    return {
        "__type": make_cls_key(type(inst)), 
        **serializers[type(inst)](inst)
    }

def deserialize(serialized):
    key = serialized.pop("__type")
    return deserializers[key](serialized)

The cattrs docs are pretty good, but it will take me a while to internalize them.

Inheritance

I would like to add prefabs and inheritance. Something like this:

>>> world = tcod.ecs.World()
>>> world["orc"].components[("atk", int)] = 4
>>> monster = world["orc"].instantiate()  # New entity with is-a relation
>>> monster.components[("atk", int)]  # Takes atk from parent
4
>>> monster.components(traverse=())[("atk", int)]  # Missing when traversal is disabled
KeyError
>>> monster.components[("atk", int)] += 1  # Typical copy-on-write
>>> monster.components(traverse=())[("atk", int)]
5

I'm in the middle of refactoring queries, and once that is done I'll be able to add relationship traversal and then I can make an IsA sentinel value which will be the default traversal option for all queries and lookups.

Ways to query or obtain all entities of a world

If I am not mistaken, there is no proper way to simply query all entities in a world, as queries require a filter to include entities.

It would be nice to be able to query all entities, whether with something like

world.Q()  # Same result as a query matching all entities

or even by allowing to iterate over the world object

for entity in world:
    pass

The lack of iterator on worlds also seems to create an infinite loop when one tries to iterate over them. Implementing an iterator or raising an error would make things clearer.

A bit further than that, this raises the question to include or not the global entity of the world. In my opinion, it seems logical that the query would not include it, while the iterator would, as the query "feels" external to the world, while the iterator is internal (someone queries about the world, but iterates on/from the world).

That's my suggestion! I hope it makes sense.

Entity Relationships

Just a quick issue in case I lose track of these articles:

Main article: https://ajmmertens.medium.com/building-games-in-ecs-with-entity-relationships-657275ba2c6c

Semi-related (Hierarchies):
https://skypjack.github.io/2019-06-25-ecs-baf-part-4/
https://skypjack.github.io/2019-08-20-ecs-baf-part-4-insights/

Compiled queries

Later I might want to speed up queries by not discarding the entities after they're used.

In theory a query could register itself with the world holding into its entities and leaving info on how adding or removing components would affect that entities placement in that query. Like saying that removing a component will always remove it from a query and adding that component might add it back into the query, so only entities which may be added will need to be checked if the query is used.

Removing entities from the World

Hello,

I was wondering if there is a way to delete/remove an entity from the world.
Typically, if I want to remove an obsolete entity, I would do a entity.clear() which would effectively make the entity "invisible" to queries, if I'm not mistaken; however, the world keeps a reference to the entity which prevents it from being deleted.

Is this intentional? Doesn't it create a memory leak where creating many entities lead to an increasing amount of references to objects that are not used anymore? Or did I miss something?

Thanks.

Numpy components

I've had theories of assigning components based on Numpy dtypes and then accessing those in a vectorized way, but it's hard to settle on how this would work in practice. The biggest issue is how everything works with sparse-sets right now but Numpy stuff would work best with archetypes. With sparse-sets all the performance benefits of Numpy are lost as it'd take so long to lookup the array indexes that you could just use Python objects directly.

EnTT may have solved this problem with "groups" but I'm not experienced enough to follow the article at the moment.
https://skypjack.github.io/2019-04-12-entt-tips-and-tricks-part-1/

Writing this down to keep track of it. I probably won't come back to this for a while.

Query by component values

It's possible to add a new lookup table to hold the (component_key, value) combination of components, this would allow querying these values directly:

@attrs.define(frozen=True)
class Position():
    x: int
    y: int

entity = world.new_entity()
entity.components[Position] = Position(1, 2)

assert set(world.Q.all_of(component_values={Position: Position(1, 2)})) == {entity}
assert set(world.Q.all_of(component_values=[(Position, Position(1, 2))])) == {entity}

It's something to keep in mind. I'm not 100% sure the added cost is worth it. There's already other methods of doing this, but this new method would be simpler.

I could add this feature and then remove it later if it doesn't work out.