Conflict Resolution Strategies

1. The Core Challenge: Conflict Detection

Before we can resolve a conflict, we must detect it. Conflicts arise when a client attempts to save changes based on an outdated version of a file. The key to detection is optimistic concurrency control.

Here's the flow:

1. When a client has local changes to upload, it sends the new data to the server.
2. Crucially, along with the data, it also sends the version_id of the file it based its changes on (let's call it base_version_id). This is the version the client had before it started making local edits.
3. The metadata service, our source of truth, looks up the current latest version_id for that file in its database.
4. Conflict Detection:
   • If the client's base_version_id matches the server's latest_version_id, there is no conflict. The server accepts the changes and creates a new version.
   • If the client's base_version_id is older than the server's latest_version_id, it means another client has synced its changes in the meantime. This is a conflict.

Now, the critical question is: what does the server do when it detects a conflict?

2. Strategy 1: The File-Sync Model (e.g., Dropbox)

For a generic file synchronization service that handles opaque files (Word docs, images, binaries), the system cannot intelligently merge the contents of the files. The primary goal is to prevent data loss and delegate the resolution to the user.

The most common and robust approach is for the client to handle the reconciliation:

1. Server Rejects: Upon detecting a conflict, the server rejects the client's upload request. It responds with an error indicating a conflict and provides the latest_version_id.
2. Client Downloads: The client application receives the conflict error. It then downloads the latest version of the file from the server.
3. Client Reconciles: The client now has two versions: the latest version from the server and the user's local, conflicting changes. Since it cannot automatically merge them, it saves the user's local version as a new, separate file. Typically, this is named something like "MyFile (Conflicted Copy).txt".
4. User Notified: The system then informs the user that a conflict occurred and that their changes were saved in a separate file, prompting them to manually merge the two documents.

This strategy is simple, safe, and guarantees no data is ever lost. It places the burden of semantic merging on the only entity that understands the file's content: the user.

An Advanced Alternative: Server-Side "Siblings"

While "reject and reconcile" is a perfectly valid and common approach, you can demonstrate deeper knowledge by discussing a more advanced server-side strategy inspired by distributed databases like DynamoDB.

To summarize the video's core idea in our file-sync context:

• Instead of rejecting the conflicting upload, the server could accept it.
• It marks the new version and the existing latest version as "siblings" or divergent branches in the file's history. The version history is no longer a straight line but a tree.
• The next time any client requests the file, the server sends back both conflicting versions and a notice that they need to be merged.
• The client application would then trigger the same reconciliation logic as before: save one version as a "conflicted copy" and ask the user to merge.

Trade-offs of the "Siblings" Approach:

• Pro: It can be more efficient by avoiding a client-side retry loop (reject -> download -> re-upload). The conflicting change is accepted on the first try.
• Con: It significantly complicates the server's data model. The file_versions table must now handle a directed acyclic graph (DAG) of versions rather than a simple linear history. This adds complexity to queries and garbage collection.

In an interview, you should present the "reject and reconcile" method first, as it's the standard. Then, introduce the "siblings" approach as a sophisticated alternative, clearly articulating its trade-offs.

3. Strategy 2: The Collaborative Editing Model (e.g., Google Docs)

What if you're not synchronizing opaque files but building a collaborative editor where multiple users edit the same document in real-time? In this case, creating "conflicted copies" is not a viable user experience. The system itself must be able to automatically merge changes at a very granular level (character by character).

This is a fundamentally different problem that requires specialized algorithms. The two dominant approaches are Operational Transformation (OT) and Conflict-free Replicated Data Types (CRDTs).

To complement this reading, let's get another perspective that dives into real-world battle scars and implementation details.

Let's synthesize the key points from these readings.

Operational Transformation (OT)

• Mechanism: Clients send operations (e.g., insert 'a' at position 5) to a central server. The server is responsible for transforming incoming operations against any concurrent operations it has already processed, ensuring that all operations are applied in a consistent order. It then broadcasts the transformed operation to all other clients.
• Requires: A central server to act as the single point of coordination.
• Pros:
  • Mature and battle-tested at massive scale (Google Docs).
  • Generally lower metadata overhead, leading to smaller message payloads.
• Cons:
  • Mathematically complex to implement correctly (many edge cases).
  • Requires a constant connection to the central server, making offline support difficult.

Conflict-free Replicated Data Types (CRDTs)

• Mechanism: The data structure itself is designed to handle conflicts. Operations can be applied in any order, and the data structure mathematically guarantees that all replicas will eventually converge to the same state. This is often achieved by giving every element (e.g., each character) a unique, immutable ID and ordering them.
• Requires: No central server is strictly necessary for merging; it can be done peer-to-peer.
• Pros:
  • Naturally supports offline work and peer-to-peer collaboration.
  • Conceptually simpler in that the merge logic is baked into the data type.
• Cons:
  • Can have significant metadata overhead. Every character might need a unique ID, timestamp, etc., and "tombstones" are needed to mark deletions, which can cause data structures to grow indefinitely if not managed.
  • Less mature ecosystem compared to OT for some applications.

When to Use Which?

• File Sync (Dropbox): Neither. These algorithms are for applications that understand and control the document's internal structure. A file sync service treats files as opaque blobs.
• Collaborative Editing (Google Docs): OT is a strong choice when you have a reliable central server, performance is critical, and offline is a secondary concern.
• Collaborative Design/Offline-First (Figma, Notion): CRDTs are ideal when offline support is a primary requirement, the application state is more complex than linear text (e.g., a graph of design objects), or you want to enable peer-to-peer sync.

Conclusion

Handling concurrency is a pivotal part of designing any distributed storage system. Failing to do so correctly leads to data loss and destroys user trust. In this lesson, we dissected the primary strategies for managing concurrent edits.

Key Takeaways:

• Detect First: Use optimistic concurrency control with version numbers (base_version_id) to reliably detect conflicts.
• For File Sync (Opaque Files): The standard, safest model is "reject and reconcile." The server rejects the conflicting write, and the client creates a "conflicted copy" for the user to merge manually.
• For Collaborative Editing (Structured Data): Use specialized algorithms.
  • Operational Transformation (OT): A centralized, server-based approach that is mature and performant. Best for always-online applications.
  • CRDTs: A decentralized approach ideal for offline-first and peer-to-peer applications, at the cost of higher metadata overhead.
• Articulate the "Why": In an interview, it's not enough to name these strategies. You must explain why you would choose one over the other based on the specific system requirements (e.g., opaque files vs. structured text, online vs. offline-first).

Next Lesson Preview:

We have now covered the core design of a file sync service, from client logic and metadata management to conflict resolution. In our next lesson, we will pivot to a different type of large-scale system: media streaming. We will begin by tackling how to design a video ingestion and processing pipeline for a streaming platform like YouTube or Netflix.