Can intelligent agents learn to cooperate?  We’ll look at this in two different settings, which we’re interested in for different reasons. First is Dec-POMDPs, where the agents share a common objective, but cooperation is still nontrivial because of incomplete information. The CIRL setup is a great example of this. Dec-POMDPs often have a trivial solution where the agents agree in advance on a communication scheme, but this is useless for human-machine coordination. We’ll look at a proposed solution in the context of Hanabi.

Next we’ll look at cooperation between agents with incompatible preferences. The most obvious way this can happen is through reciprocity; we’ll see this in the context of LOLA.

Slides

L04_Cooperation.pdf

Student Presentation 1 (zero-shot coordination)

Student Presentation 2 (LOLA)

Alignment Agendas

• Dafoe et al., 2020. Open problems in cooperative AI

Paper Presentations

• Cui et al., 2021. K-level reasoning for zero-shot coordination in Hanabi
• Foerster et al., Learning with opponent learning awareness

Supplemental Readings

• Critch et al., 2022. Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory
• Oesterheld et al., 2023. Similarity-based cooperative equilibrium