Secure Credential Management with AWS Secrets Manager

1. The Problem with Hardcoded Secrets

As a Data Engineer, your ETL jobs and applications frequently need access to sensitive information:

• Database connection strings (usernames and passwords)
• API keys for third-party services
• OAuth tokens

Hardcoding this information into your source code or configuration files poses several risks:

• Exposure: If your code is checked into a version control system like Git, even in a private repository, the secrets are exposed to anyone with access.
• Operational Overhead: Changing a password requires finding every instance of the hardcoded secret, updating it, and redeploying the application. This is slow and error-prone.
• Lack of Auditability: You have no easy way to know who or what is accessing a secret.

AWS Secrets Manager is designed to solve exactly these problems.

2. Introducing AWS Secrets Manager

AWS Secrets Manager is a service that helps you protect the secrets needed to access your applications, services, and IT resources. It enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

The core idea is simple: instead of embedding secrets in your code, you store them in Secrets Manager and have your application retrieve them programmatically via an API call at runtime.

To get a clear picture of what Secrets Manager is and how it works, let's watch a short video.

As the video explained, three AWS services work together to secure your secrets:

1. Secrets Manager: Stores the secret (as key-value pairs or plaintext).
2. AWS KMS: Encrypts the secret at rest. You can use an AWS-managed key or a customer-managed key (CMK) for more control.
3. AWS IAM: Controls which principals (users, roles, or services like AWS Glue and Lambda) are permitted to retrieve the secret.

What to Store (and What Not to Store)

Secrets Manager is versatile, but it's important to use it for the right purpose. It's ideal for:

• Database credentials (for RDS, Redshift, or on-premises databases)
• Third-party API keys (e.g., Stripe, GitHub)
• SaaS application credentials
• Any custom secret in plaintext or JSON format

However, for some types of secrets, other AWS services are a better fit:

• AWS credentials: Use IAM roles instead of storing access keys.
• Encryption keys: Use AWS Key Management Service (KMS).
• SSL/TLS certificates: Use AWS Certificate Manager (ACM).

3. Secrets Manager vs. Systems Manager Parameter Store

For the exam, you must be able to differentiate Secrets Manager from another AWS service that can store secrets: AWS Systems Manager Parameter Store. Both can store sensitive strings, but they are designed for different use cases and have different cost structures.

The key takeaway is to choose the service based on your needs:

• Use Secrets Manager when you need automatic password rotation or are managing your most critical credentials.
• Use SSM Parameter Store for general configuration data, feature flags, or less sensitive secrets to optimize costs.

This video offers a great breakdown of this decision-making process.

4. Implementing Secrets Manager in a Data Pipeline

Now let's focus on the "implement" part of our learning outcome. In a typical data engineering scenario, an AWS Glue job or a Lambda function needs credentials to connect to a data source or sink.

Here is a common architecture for an ETL pipeline:

To make this work, you need to:

1. Store the secret in AWS Secrets Manager.
2. Create an IAM Role for your Glue job or Lambda function.
3. Attach a policy to the role granting it permission to retrieve the secret.
4. Write code in your script to call the Secrets Manager API and fetch the secret.

Step 1 & 2: Storing the Secret and Creating the Role

You can create a secret easily via the AWS Console or CLI. When you create it, you typically provide a name and the secret value as key-value pairs.

Next, you create an IAM Role that your service (e.g., AWS Glue) will assume.

Step 3 & 4: Granting Permissions and Retrieving the Secret

The IAM role needs a policy that allows it to perform the secretsmanager:GetSecretValue action. If your secret is encrypted with a customer-managed KMS key (CMK), the role will also need kms:Decrypt permissions on that key, linking back to our previous lesson.

Here is a sample Python code snippet using the boto3 library, which you would include in your Glue or Lambda script to retrieve the secret:

import boto3
import json

def get_secret(secret_name, region_name):
    """
    Retrieves a secret from AWS Secrets Manager.
    """
    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    try:
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
    except Exception as e:
        # Handle exceptions here
        print(f"Error retrieving secret: {e}")
        raise e
    else:
        # Decrypts secret using the associated KMS key.
        # Depending on whether the secret is a string or binary, one of these fields will be populated.
        if 'SecretString' in get_secret_value_response:
            secret = get_secret_value_response['SecretString']
            return json.loads(secret)
        else:
            # Handle binary secret if needed
            decoded_binary_secret = base64.b64decode(get_secret_value_response['SecretBinary'])
            return decoded_binary_secret

# --- Example Usage in your ETL script ---
secret_name = "prod/my-data-warehouse/creds"
region = "us-east-1"

# Fetch the credentials
credentials = get_secret(secret_name, region)
username = credentials['username']
password = credentials['password']

# Now use the username and password to connect to your database
# ... database connection logic ...

This approach completely decouples the secret from the code. If the password changes (either manually or through automatic rotation), you only update it in Secrets Manager, and the code will fetch the new value on its next run without any changes or redeployment.

For a practical guide with CLI commands and more code examples, you can refer to the following resource.

Conclusion

You have now learned how to handle one of the most common security challenges in application development and data engineering. By mastering AWS Secrets Manager, you can build data pipelines that are not only functional but also secure, scalable, and compliant.

Key Takeaways:

• Never hardcode secrets. Always store them in a dedicated secrets management service.
• AWS Secrets Manager securely stores, encrypts, and manages the lifecycle of credentials for your data pipelines.
• Choose between Secrets Manager and SSM Parameter Store based on the need for automatic rotation and cost considerations.
• Access to secrets is controlled by IAM permissions (secretsmanager:GetSecretValue) and KMS permissions (kms:Decrypt).
• In your data pipelines, services like AWS Glue and Lambda use an IAM role to retrieve credentials programmatically at runtime.

In our next lesson, we will shift our focus from protecting data and credentials to securing the network itself. We will configure VPC endpoints for private connectivity to AWS data services, ensuring that traffic between your applications and services like S3 never leaves the AWS network.