Deep Learning

Represent a feedforward neural network as a composition of affine transformations and nonlinear activations.

Apply the chain rule to trace how backpropagation computes a parameter gradient.

Select an output activation and loss function for a given regression or classification task.

Choose activation functions and initialization schemes that support stable gradient flow.

Distinguish the update behavior of stochastic gradient descent and Adam.

Fit a multilayer perceptron to a nonlinear function-approximation task.

Diagnose underfitting, overfitting, vanishing gradients, and exploding gradients from training evidence.

Identify the agent, environment, state, action, reward, and termination signals in a reinforcement-learning problem.

Represent a sequential decision problem as a Markov decision process.

Compute discounted returns from a finite trajectory.

Distinguish a policy, state-value function, action-value function, and advantage function.

Use Bellman expectation equations to relate values across successive states.

Use Bellman optimality equations to characterize optimal value functions.

Interact with a Gymnasium environment while handling termination and truncation correctly.

Evaluate a fixed policy using iterative dynamic programming.

Compute an improved policy from an action-value function.

Find an optimal policy using value iteration.

Estimate a state-value function from complete episodes with Monte Carlo prediction.

Estimate a state-value function using one-step temporal-difference learning.

Distinguish on-policy SARSA updates from off-policy Q-learning updates.

Implement tabular Q-learning with an epsilon-greedy behavior policy.

Assess whether a tabular agent has learned using return curves and repeated evaluation episodes.

Explain why neural function approximation is needed when state spaces are too large for tables.

Construct a Q-network whose outputs represent action values for a discrete action space.

Implement an epsilon schedule for deep value-based exploration.

Store and sample transitions with an experience replay buffer.

Compute a bootstrapped DQN target using a separate target network.

Implement one DQN optimization step in PyTorch.

Assemble a complete DQN training loop for a Gymnasium environment.

Diagnose DQN learning failures using returns, Q-values, losses, and gradient statistics.

Explain how correlated data, moving targets, and maximization bias can destabilize DQN.

Implement Double DQN target computation to reduce overestimation bias.

Construct a dueling network that separately estimates state value and action advantages.

Compute multi-step return targets for value learning.

Apply prioritized replay with importance-sampling correction.

Design a controlled ablation that isolates the effect of one DQN enhancement.

Evaluate a deep-RL agent across multiple random seeds with uncertainty summaries.

Parameterize a categorical policy for a discrete action space in PyTorch.

Compute log-probabilities and entropy from a PyTorch policy distribution.

Explain the likelihood-ratio objective underlying the policy-gradient estimator.

Compute a Monte Carlo policy-gradient loss from sampled trajectories.

Use a learned baseline to reduce policy-gradient variance without introducing action-dependent bias.

Add entropy regularization to encourage exploration.

Implement and evaluate REINFORCE with a value-function baseline.

Compute a one-step temporal-difference advantage estimate.

Construct separate actor and critic losses from a batch of transitions.

Implement an on-policy actor–critic update in PyTorch.

Explain the bias–variance trade-off controlled by multi-step advantage estimates.

Compute generalized advantage estimates from rewards, values, and episode boundaries.

Normalize advantages without mixing information across invalid episode boundaries.

Collect synchronous rollouts from vectorized environments for actor–critic training.

Compute the probability ratio between current and behavior policies.

Explain how PPO's clipped surrogate objective limits destructive policy updates.

Compute PPO policy, value, and entropy-loss terms from a rollout batch.

Organize rollout data into shuffled minibatches for repeated PPO updates.

Implement a single clipped PPO optimization epoch in PyTorch.

Assemble an end-to-end PPO trainer with generalized advantage estimation.

Monitor approximate KL divergence and clipping fraction during PPO training.

Tune one PPO hyperparameter through a controlled experiment rather than trial-and-error changes.

Represent a continuous-action policy with a diagonal Gaussian distribution.

Apply a tanh transformation to bound sampled actions to an environment's valid range.

Correct a Gaussian log-probability for the tanh transformation.

Distinguish deterministic actor–critic methods from stochastic actor–critic methods.

Explain how entropy maximization changes the reinforcement-learning objective.

Compute critic targets for Soft Actor–Critic using target critics and sampled next actions.

Compute the Soft Actor–Critic policy loss and temperature loss.

Implement and evaluate the core update steps of Soft Actor–Critic in PyTorch.

Identify a paper's research question, claimed contribution, and supporting evidence.

Translate a deep-RL paper's notation into consistent definitions of states, actions, returns, values, and objectives.

Trace the dependencies among equations used to define a paper's learning algorithm.

Convert algorithm pseudocode into an ordered list of implementable components.

Infer tensor shapes and gradient-flow boundaries from mathematical descriptions.

Distinguish essential algorithm details from experimental and engineering choices.

Evaluate whether baselines, ablations, metrics, and seed counts support a paper's claims.

Locate implementation-critical details across the main text, appendices, and cited prior work.

Define a reproducible target claim from a selected deep-RL paper.

Write an implementation specification that maps each paper equation to a software component.

Create minimal unit tests for returns, targets, losses, and boundary handling before training.

Match a paper's environment preprocessing, evaluation protocol, and reporting metric.

Build a configuration that records hyperparameters, seeds, software versions, and environment versions.

Use intermediate diagnostics to localize discrepancies between an implementation and a paper.

Compare reproduced results with reported results using uncertainty rather than a single training run.

Document implementation deviations, negative results, and remaining uncertainties in a concise reproduction report.