EMRFS S3 Integration and Encryption

1. EMRFS: Decoupling Compute and Storage

The magic that allows EMR to use S3 as if it were a local file system is the EMR File System (EMRFS). EMRFS is a special implementation of the Hadoop Distributed File System (HDFS) that connects directly to S3.

This architecture provides a powerful advantage: the decoupling of compute and storage. Your data can live persistently and cost-effectively in S3, while your EMR clusters—the compute layer—can be created on-demand for specific jobs and terminated when finished to save costs.

Before your EMR cluster can interact with S3, it needs permission. This is granted via an IAM role attached to the EC2 instances that make up the cluster, known as an EC2 instance profile. The default role, EMR_EC2_DefaultRole, typically has the necessary S3 permissions, but for production environments, you should always follow the principle of least privilege.

The video below briefly touches on the importance of this instance profile.

2. Encryption for Data at Rest in S3

Since you're using S3 as the primary data lake storage for EMR, ensuring that data is encrypted at rest is a critical security requirement. EMRFS supports multiple encryption options, which can be categorized into two main types:

• Server-Side Encryption (SSE): You upload an object to S3, and S3 encrypts it after receiving it, before saving it to disk.
• Client-Side Encryption (CSE): The EMR cluster (the "client") encrypts the data before uploading it to S3. The data is never transmitted over the network in plaintext.

The modern and recommended way to manage these settings for EMR is through a Security Configuration. This is a reusable set of security options that you can create once and apply to any number of clusters.

3. Implementing Encryption with a Security Configuration

Let's walk through how to create and configure a security configuration for S3 encryption.

As you read, you'll see the console presents four choices for S3 encryption. Let's clarify what they mean:

• SSE-S3: The simplest option. S3 manages the encryption keys for you. It provides a baseline level of encryption with minimal configuration.
• SSE-KMS: S3 still performs the encryption, but it uses a key that you manage in AWS Key Management Service (KMS). This is a common choice as it gives you control over the key's policies and provides an audit trail of its usage via CloudTrail.
• CSE-KMS: The EMR cluster encrypts the data before sending it to S3, using a key managed in AWS KMS. This ensures end-to-end encryption from your cluster to the S3 service.
• CSE-Custom: The EMR cluster encrypts the data using a custom key provider you supply (e.g., a hardware security module). This offers the most control but also the most management overhead.

The Legacy Method: emrfs-site Properties

Before security configurations were introduced (in EMR release 4.8.0), encryption was configured using emrfs-site properties passed to the cluster at launch. While security configurations are now the best practice, you may still encounter this method in older setups or exam questions.

4. Step-by-Step Configuration Workflow (Using KMS)

Let's synthesize this into a practical workflow for enabling S3 encryption with a customer-managed KMS key, which is a very common pattern for the exam.

1. Create a KMS Key: In the AWS KMS console, create a symmetric KMS key in the same region as your EMR cluster and S3 bucket.

2. Grant Permissions to the EMR Role: This is a crucial and often-missed step. The EC2 instance profile role used by your cluster (e.g., EMR_EC2_DefaultRole) must be given permission to use the KMS key. The easiest way to do this is to edit the key policy in the KMS console and add the EMR_EC2_DefaultRole as a "Key User".

3. Create a Security Configuration:

   • Navigate to the EMR console and select "Security configurations".
   • Click "Create security configuration".
   • Give it a name (e.g., s3-sse-kms-config).
   • Under "S3 encryption", select SSE-KMS or CSE-KMS as the mode.
   • For the AWS KMS key, select the key you created in Step 1.
   • Click "Create".

4. Launch the EMR Cluster:

   • When you create your EMR cluster, navigate to the "Security configuration and permissions" section.
   • In the Security configuration dropdown, select the configuration you just created (s3-sse-kms-config).
   • Ensure the Permissions are set to use an instance profile role that has been made a Key User (e.g., EMR_EC2_DefaultRole).

Now, any data that your EMR jobs write to S3 will be automatically encrypted according to the policy defined in your security configuration.

Conclusion

In this lesson, we bridged the gap between your EMR cluster and your S3 data lake. You learned how EMRFS enables the powerful decoupling of compute and storage and, most importantly, how to secure the data EMR writes to S3 using encryption.

Key Takeaways:

• EMRFS is the connector that allows EMR to use S3 as its primary, persistent file system.
• Security Configurations are the modern, recommended best practice for managing encryption and authentication settings for EMR clusters.
• You must choose between Server-Side Encryption (SSE) and Client-Side Encryption (CSE) based on your security requirements. SSE-KMS and CSE-KMS are common choices that provide auditability and control.
• A critical configuration step is ensuring the EC2 instance profile for your EMR cluster has IAM permissions to use the specified AWS KMS key.

Next Up:
We've now configured the cluster's compute resources, secured its network, and established a secure connection to its data store. In the next lesson, we'll finally put it to work! We will learn how to "Write a Spark application using PySpark to perform common data transformations (e.g., select, filter, join, groupBy)." This will be our first step into the application layer, where you'll see how data is actually processed on the cluster.