Designing Cross-Account Data Sharing with Lake Formation

Why Traditional Cross-Account Sharing Is Hard

Before Lake Formation, sharing data across accounts typically involved creating complex S3 bucket policies and cross-account IAM roles. This approach had several drawbacks:

• Complexity: IAM policies can become long and difficult to manage, especially with many accounts and tables.
• Error-Prone: A small mistake in a policy could either block access or grant overly permissive access.
• Lack of Discoverability: A data analyst in a consumer account had no easy way to know what data was shared with them.
• No Fine-Grained Control: It was difficult to enforce column or row-level security using just IAM and S3 policies.

Lake Formation was built to solve these problems by providing a centralized, simplified, and secure way to manage data access, both within and across accounts.

The Core Mechanics of Lake Formation Sharing

To understand how Lake Formation works its magic, we need to grasp two key concepts: AWS Resource Access Manager (RAM) and Resource Links.

The following video provides an excellent overview of the business motivations and the technical foundation of the feature.

Let's summarize the key takeaways from the video:

1. AWS Resource Access Manager (RAM): This is the underlying AWS service that facilitates sharing resources between accounts. When you grant access in Lake Formation to an external account, Lake Formation uses RAM to create a "resource share." The consumer account must then accept this share in the RAM console.

   • Important Note: If both accounts are part of the same AWS Organization, you can configure RAM to automatically accept shares, simplifying the process. For external accounts, manual acceptance is required.

2. Resource Links: A shared database or table is not directly visible or usable in a consumer account's query engine like Athena. The consumer's Data Lake Administrator must first create a Resource Link. A resource link is essentially a pointer or a symbolic link in the consumer's local Glue Data Catalog that points to the shared resource in the producer's catalog.

   • This solves two problems: it makes the shared resource discoverable in local tools, and it prevents naming collisions if multiple accounts share resources with the same name.

The Cross-Account Sharing Workflow

Now let's walk through the end-to-end process. It involves two parties: the producer account (owner of the data) and the consumer account (recipient of the share).

This process is demonstrated in detail in the video you just watched, but the following article provides a concise, step-by-step summary that is easy to follow.

Summary of the Process:

In the Producer (Source) Account:

1. Grant Permissions: In Lake Formation, select the database or table to share.
2. Choose Grant and select External account.
3. Enter the 12-digit AWS Account ID of the consumer.
4. Select the permissions to grant (e.g., SELECT, DESCRIBE).
5. Crucially, decide whether to make the permissions Grantable. If you select Grantable, you allow the Data Lake Admin in the consumer account to further grant those permissions to principals (users and roles) within their own account. This is the standard practice for delegating management.

In the Consumer (Target) Account:

1. Accept Share: Go to the AWS RAM console and accept the resource share invitation sent from the producer account.
2. Create Resource Link: In the Lake Formation console, you will now see the shared resource. Select it and choose Create resource link. Give the link a name and choose a local database to store it in.
3. Grant Local Permissions: The consumer's Data Lake Admin can now grant permissions on the resource link to local IAM users and roles.
4. Query Data: The local users can now query the resource link in Amazon Athena as if it were a native table in their account. All query costs are incurred by the consumer account.

Architectural Design Patterns

Now that we understand the mechanics, let's zoom out to the architectural level. For the exam, you'll need to recommend a sharing pattern based on an organization's requirements for governance and autonomy. There are two primary patterns, plus a hybrid.

This AWS documentation page provides clear definitions of these patterns.

Here's a breakdown of the patterns:

1. Peer-to-Peer (Data Mesh)

• Concept: A decentralized model where different business units (data producers) own their data and are responsible for sharing it directly with consumers. There is no central governing body acting as a gatekeeper.
• Analogy: Think of it as a free market. If you need data from the Marketing department, you request it directly from them.
• Pros: High autonomy and agility for producers. Reduces bottlenecks.
• Cons: Can be difficult to enforce consistent governance standards and discover data across the organization.
• Best for: Organizations with a culture of high trust and autonomy between business units, or where speed and independence are prioritized over centralized control.

2. Hub-and-Spoke

• Concept: A centralized model. A single "hub" account owns the central data catalog and often the data itself. Other "spoke" accounts act as producers who write data to the hub, and consumers who read data from it. The central governance team in the hub account controls all access.
• Analogy: A central library. All books (data) are stored in one place, and the head librarian (central team) decides who can check them out.
• Pros: Strict, centralized governance, security, and auditing. Easy data discovery.
• Cons: Can become a bottleneck, slowing down access to data. Less autonomy for individual teams.
• Best for: Organizations in highly regulated industries (like finance or healthcare) that require tight, auditable control over all data assets.

3. Centrally Governed Data Mesh (Hybrid)

• Concept: This model combines the best of both worlds. Producers still own and manage their data in their own accounts (like a data mesh). However, they "publish" their data's metadata to a central governance account. This central catalog enables discovery and auditing across the organization. Access requests might be routed through the central team, but producers often still grant the final permissions.
• Analogy: A modern marketplace like Amazon. Individual sellers (producers) own their products, but they list them on a central platform where customers (consumers) can discover them and where the platform enforces certain rules.
• Pros: Balances producer autonomy with centralized discovery, auditing, and governance. Highly scalable.
• Cons: More complex to set up than the other two models.
• Best for: Large, complex enterprises that want the scalability of a data mesh but still need strong central governance and a unified view of their data assets.

For a real-world perspective on why a large enterprise would choose a data mesh architecture, let's hear from JP Morgan Chase.

Conclusion

In this lesson, we have demystified cross-account data sharing in AWS. By leveraging Lake Formation, you can move beyond cumbersome IAM policies and build secure, scalable, and manageable data sharing solutions.

Key Takeaways:

• Lake Formation uses AWS RAM to orchestrate resource sharing and requires consumers to create Resource Links to access shared data.
• The Grantable permission is essential for delegating permission management to consumer accounts.
• You can design sharing architectures using three main patterns:
  • Peer-to-Peer (Data Mesh): For maximum autonomy.
  • Hub-and-Spoke: For maximum central control.
  • Centrally Governed Data Mesh (Hybrid): A scalable balance of both.
• Choosing the right pattern depends on an organization's culture, governance requirements, and scale.

Next Up:
We have now established a comprehensive governance framework for securing data both within and across accounts. The next logical step is to automate the pipelines that process and transform this data. In our next module, we will dive into Data Pipeline Orchestration, where we'll learn to use services like AWS Step Functions and Amazon EventBridge to build robust, event-driven workflows.