Reinforcement Learning

Represent a sequential decision problem as a finite Markov decision process by specifying its states, actions, transitions, rewards, and discount factor.

Compute discounted returns and state-action visitation frequencies from sampled trajectories.

Derive the Bellman expectation equations for a fixed policy.

Evaluate a policy in a finite Markov decision process using dynamic programming.

Compute an optimal policy in a finite Markov decision process using value iteration.

Implement on-policy temporal-difference control with SARSA and epsilon-greedy exploration.

Implement off-policy temporal-difference control with Q-learning.

Mini-project: compare SARSA and Q-learning in a stochastic gridworld using controlled learning curves.

Explain how experience replay and target networks stabilize deep Q-learning.

Implement a Deep Q-Network update with replay sampling and a target network.

Derive the likelihood-ratio policy-gradient estimator from the expected-return objective.

Implement REINFORCE with a learned value baseline.

Compute generalized advantage estimates from a batch of trajectories.

Derive the actor, critic, and entropy terms of an advantage actor-critic objective.

Implement the clipped PPO policy and value updates in PyTorch.

Mini-project: run a clipping-and-entropy ablation for PPO and diagnose its learning curves.

Formulate a multi-agent task as a partially observable stochastic game.

Represent a cooperative task as a decentralized partially observable Markov decision process.

Calculate centralized and decentralized value functions for a small cooperative game.

Explain why simultaneous policy learning makes each agent's effective environment non-stationary.

Identify coordination, exploration, and credit-assignment failures in multi-agent trajectories.

Explain the information available during training and execution under centralized training with decentralized execution.

Determine whether a value factorization satisfies the individual-global-max consistency requirement.

Mini-project: train independent PPO agents in a PettingZoo task and quantify variation across random seeds.

Implement value decomposition networks with additive per-agent utilities.

Explain how QMIX's monotonic mixing constraint enables decentralized greedy action selection.

Implement a QMIX mixing network conditioned on global state.

Derive the counterfactual multi-agent advantage used by COMA.

Explain how MADDPG uses centralized critics with decentralized actors.

Implement a centralized value critic for MAPPO.

Select an appropriate cooperative MARL algorithm from a task's action space, observability, and coordination structure.

Mini-project: benchmark independent learning, VDN, QMIX, and MAPPO on one compact cooperative task.

Compute pure and mixed Nash equilibria in two-player matrix games.

Distinguish Nash efficiency, Pareto efficiency, and collective return in cooperative and mixed-motive games.

Diagnose social-dilemma incentives from individual and collective payoff structures.

Explain how repeated interaction can support reciprocity and conditional cooperation.

Implement a difference reward for assigning credit in a team task.

Train a bandwidth-limited differentiable communication channel between agents.

Measure the causal contribution of learned messages by intervening on the communication channel.

Mini-project: test whether communication or shaped credit improves cooperation in a sequential social dilemma.

Distinguish self-play, cross-play, zero-shot coordination, and ad hoc teamwork evaluation protocols.

Construct and interpret a cross-play matrix for a population of independently trained policies.

Diagnose convention overfitting from self-play and cross-play results.

Generate a behaviorally diverse training population using a diversity objective.

Explain how symmetry-aware other-play reduces dependence on arbitrary conventions.

Infer a compact representation of a partner from its observed behavior.

Compare zero-shot performance with few-shot online adaptation to a new partner.

Mini-project: evaluate a cooperative policy against held-out partner populations with distinct conventions.

Classify a distribution shift as a change in observations, dynamics, rewards, tasks, or partners.

Design separate training, validation, and test distributions for a MARL benchmark.

Generate procedurally varied cooperative tasks from parameterized environment rules.

Apply domain randomization to train policies across environment variations.

Train a policy with a robust objective that emphasizes poorly performing environment variants.

Explain the inner and outer optimization loops of meta-reinforcement learning.

Report out-of-distribution performance with seed-level uncertainty and aggregate generalization metrics.

Mini-project: evaluate trained teammates on held-out layouts and task parameters without retraining.

Distinguish open-ended learning from fixed-task training using novelty, diversity, and progress criteria.

Implement an automatic curriculum based on agents' learning progress.

Explain minimax environment generation in PAIRED-style training.

Maintain a quality-diversity archive indexed by agent or environment behavior.

Design temporally extended skills for a long-horizon cooperative task.

Construct a recurrent belief-state representation from partial observation histories.

Ground language-level goals in executable actions within an embodied simulator.

Mini-project: generate and solve a small curriculum of procedurally varied embodied cooperation tasks.

Interpret an autoregressive language model as a stochastic policy over tokenized actions.

Constrain an LLM agent to valid environment actions using schemas and tool interfaces.

Implement an observation-to-action LLM agent loop with structured outputs.

Add episodic retrieval memory to an LLM agent and measure its effect on task completion.

Apply inference-time search or self-consistency to improve an LLM agent's decisions.

Distinguish supervised fine-tuning, preference optimization, and reinforcement learning for adapting LLM agents.

Evaluate an LLM agent's success, cost, latency, variance, and action validity.

Mini-project: build and evaluate a tool-using LLM agent in an interactive text environment.

Compare centralized, decentralized, and hierarchical control architectures for teams of LLM agents.

Design a token-budgeted protocol for inter-agent communication and shared state.

Represent joint intentions, role assignments, and commitments in a machine-readable team plan.

Infer another agent's goal or role from its messages and actions.

Combine an LLM high-level planner with learned low-level MARL policies.

Train or prompt a population of LLM agents with diverse cooperative strategies.

Test a multi-agent LLM system for collusion, deception, reward hacking, and communication failures.

Mini-project: build an LLM-agent team that coordinates with an unfamiliar partner under task variation.

Conduct a structured literature search on generalizable cooperation in MARL and LLM agents.

Classify frontier systems by whether LLMs serve as agents, planners, world models, reward models, or data generators.

Identify an unresolved research gap by comparing assumptions, evidence, and limitations across related papers.

Translate a research gap into a falsifiable hypothesis and measurable predictions.

Choose baselines and ablations that isolate the causal contribution of a proposed method.

Design a multi-seed experiment with paired comparisons, confidence intervals, and predefined exclusion rules.

Create a reproducible experiment configuration with versioned code, dependencies, data, and logging.

Mini-project: reproduce one lightweight result from a MARL or LLM-agent paper and document any discrepancy.

Write a one-page specification for a scoped study of cooperation across unseen tasks and unfamiliar partners.

Define benchmark interfaces and splits that support both learned MARL policies and LLM-based agents.

Create a reproducible repository skeleton that reuses agents, environments, and evaluation code from earlier projects.

Implement one minimal novel cooperation mechanism by composing previously tested components.

Specify and launch an experiment matrix covering baselines, ablations, held-out conditions, and random seeds.

Analyze experiment outputs using cross-play matrices, uncertainty estimates, and failure-mode categories.

Write a paper-style results section that separates supported findings from speculation.

Package the system as a runnable public artifact with setup instructions, evaluation scripts, limitations, and responsible-use notes.