mnoukhov / emergent-compete Goto Github PK

To test the usefulness of the state "context", we have a receiver agent (e.g. deterministic) paired with a sender that simply sends it's input as a message. Therefore the receiver is just trying to guess the bias based off of repeated interactions and with gradient-based methods should get 100% accuracy on the later rounds.

current situation with deterministic, bias [0,10] shows that it is not near 100% accuracy

but that the current state is indeed helpful to some extent (rewards per round shown here)

need to change:

• the state
• how states are accumulated

look into:

• better loss function?
  (cos has bad gradients at certain points)

implement MADDPG (https://arxiv.org/abs/1706.02275) to train the agents

Following #1, no try to achieve 100% accuracy without being given the extra state variable prev_diff (previous signed distance to target). Instead use previous reward states to infer the same information.

Since only one previous reward state does not give you this information because it is unsigned (e.g. guess = 7, distance = 1 means the true reward was either 6 or 8) use two previous states which should allow to disambiguate

Does a protocol always emerge in the biased game when we expect it to?

We expect communication to emerge when the reward with communication > reward without communication. In our specific game this translates to bias < 2 pi aka not fully competitive

Show that changing from cooperative to non-cooperative causes language drift

Both agents with a simple deterministic network do not manage to converge even with bias=0

the current format of actions and loss is:

• an actions and targets correspond to a points on the edge of a circle
• the loss function is the cosine of the angle between the action and the target

the issue is that the output of the network can be arbitrarily large or small, and the loss for some prediction x is the same as the loss for x + k*2pi for any integer k. This means that the gradient can go towards one minima for some input x and towards a different minima for another similar input x+eps.

one way to fix this has been to initialize the network to strictly positive values that guarantee all outputs are in the range [0, circumference] but this will be a lot harder if the model is larger, so perhaps it makes sense to change the loss function or output to be resilient to this

ideas:

• sigmoid(output) * (circumference + 2k) - k so that all necessary output points are k from being output infinity

multi-agent deterministic - deterministic with non-zero bias and previous two states, following up on #6

now that #3 is solved for a deterministic policy learned with backprop, try to do the same thing learning with RL

current A2C method only matches one-shot baseline


but results across rounds look promising, test some hyperparameter tuning and see if we can't improve performance, otherwise maybe switch to DDPG?

It seems that the right way of logging with tensorboard could be

1. separate writers for each agent with different comments (/agent_name/)
2. custom scalars for logging multiple round rewards on the same chart

currently not updating live for unknown reasons