Designing a File Sync Service: Architecture & APIs

1. Starting the Interview: Defining the System

Before writing any code or drawing any boxes, the first and most critical step in a system design interview is to clarify the requirements. As a senior engineer, this is where you demonstrate your ability to understand the product, define the scope, and avoid designing the wrong system. A vague prompt like "Design Dropbox" is an invitation to ask questions.

Let's watch how an experienced engineer breaks down the requirements for this exact problem.

To summarize and solidify these points, let's lay them out.

Functional Requirements (The Core Features)

Based on the video and common usage, a Minimum Viable Product (MVP) for a file sync service should support:

• File Upload: A user can upload a file from a client.
• File Download: A user can download a file to a client.
• Automatic Sync: Changes made on one client (e.g., adding a file to a local "Dropbox" folder) are automatically propagated to the cloud and other connected clients.
• (Optional but common) File Sharing: A user can share a file with another user.

We'll explicitly keep features like file versioning, conflict resolution, and in-browser editing out of scope for now. They are excellent topics for deep dives, which we will cover in subsequent lessons.

Non-Functional Requirements (The '-ilities')

These define the system's quality and constraints, and they are what truly shape the architecture for a senior-level design.

• High Availability: The system must be available for uploads and downloads even if some components fail. The video makes a crucial point: we should prioritize availability over strong consistency. It's acceptable if a file uploaded in one region takes a few seconds to appear in another.
• Durability: Uploaded files must never be lost. The system must be highly reliable.
• Low Latency: Upload and download times should be as fast as possible for a good user experience.
• Large File Support: The system should handle very large files (e.g., up to 50 GB, as specified in the video). This single requirement has major implications for our design.
• Scalability: The system must handle a growing number of users, files, and requests.

With the scope now clearly defined, we can move on to the next logical step: defining the system's interface.

2. Designing the Service Contract: Core Entities and APIs

With our requirements in hand, we can define the data we'll be working with and the API endpoints that clients will use.

First, let's identify the core data entities:

1. User: The account holder.
2. File: The raw binary data of the file itself.
3. FileMetadata: Information about the file, such as fileName, size, mimeType, ownerId, createdAt, updatedAt, and a pointer to where the actual file content is stored.

Separating File content from FileMetadata is a fundamental architectural decision. File content (blobs) and metadata have very different characteristics and access patterns, suggesting they should be stored and managed separately.

Now, let's design a simple API based on our functional requirements.

Here’s a summary of the initial API design:

• Upload a file: POST /files
  • Request Body: Contains the file content and its metadata.
  • Response: 200 OK on success.
• Download a file: GET /files/{fileId}
  • Response: Returns the file content and its metadata.
• Check for changes: GET /changes?since={timestamp}
  • Response: A list of FileMetadata for files that have been created or modified since the given timestamp. This endpoint is crucial for the sync functionality.

This simple set of APIs forms the contract between our client applications and the backend.

3. High-Level Architecture: The Blueprint

Now we'll sketch the high-level architecture. The goal is to create a diagram with the major components and show how they interact to fulfill the API contracts we just defined.

The main components are:

• Client: The application running on a user's machine (desktop, mobile). It has a local folder, a local database to track file state, and logic to sync with the backend.
• Load Balancer / API Gateway: The single entry point for all client requests. It handles routing, authentication, and rate limiting.
• File Service: A backend microservice that handles business logic related to metadata. It does not handle the raw file bytes itself.
• Sync Service: A backend microservice responsible for handling the logic of file synchronization and notifying clients of changes.
• Metadata Database: A database (e.g., DynamoDB, PostgreSQL) that stores the FileMetadata. This database is optimized for fast lookups and queries on file attributes.
• Blob Storage: A highly scalable and durable object storage system (like Amazon S3 or Google Cloud Storage) that stores the raw file content. We will use this as a black box and not design it ourselves.

Let's see how these components work together for the core flows.

Upload and Download Flow

Based on the video and the diagram above, the flows are:

Simplified Upload Flow:

1. Client sends an upload request to the API Gateway.
2. The request is routed to the File Service.
3. The File Service splits the request:
   a. It uploads the raw file content to Blob Storage (S3).
   b. After the file is stored, S3 returns a unique URL/key.
4. The File Service writes the file's metadata (name, size, S3 key, etc.) to the Metadata DB.
5. The service returns a success response to the client.

Download Flow:

1. Client requests to download a file using its fileId.
2. The request hits the File Service.
3. The service reads the file's metadata from the Metadata DB using the fileId.
4. The metadata contains the S3 key. The service can now generate a temporary, secure link (a pre-signed URL) to the file in S3.
5. The service returns this pre-signed URL to the client.
6. The client uses this URL to download the file directly from S3, which is far more efficient than proxying the file through our service.

The Synchronization Flow

The automatic sync is the most complex part. It involves intelligence on both the client and server.

Synchronization works in two directions:

1. Local to Remote:

   • The client application continuously watches the local sync folder for changes (file added, modified, deleted) using OS-level APIs (like FileSystemWatcher on Windows or FSEvents on macOS).
   • When a change is detected, the client initiates the upload flow for the new or modified file.

2. Remote to Local:

   • The client periodically calls the GET /changes endpoint on our Sync Service.
   • The Sync Service queries the Metadata DB to find files that have changed since the client's last sync time.
   • The client receives a list of changed files. It compares this with its local state (tracked in its own local DB) and initiates the download flow for any files it needs to update.

This high-level architecture provides a solid foundation. It separates concerns effectively and outlines the primary data flows. In a real interview, this is the point where the interviewer would start asking you to go deeper into specific areas, guided by the non-functional requirements.

For example, our simple POST /files upload API will not work for a 50 GB file due to request size limits and the poor user experience of a single, long-running HTTP request. This observation naturally leads to a deep dive on file chunking, which is the topic of our next lesson.

Conclusion

In this lesson, we created the high-level design for a file synchronization service. We followed a structured interview approach, moving from requirements to APIs to the architectural blueprint.

Key Takeaways:

• Structured Approach: A system design problem should be tackled methodically: clarify functional and non-functional requirements, define core entities and APIs, and then sketch the high-level architecture.
• Separation of Concerns: A critical architectural pattern is to separate file metadata from the actual file content (blobs). This allows you to use the right storage technology for each: a fast, queryable database for metadata and a cheap, scalable object store for blobs.
• Core Components: The high-level design consists of a smart Client, stateless backend Services (for file and sync logic), a Metadata DB, and Blob Storage.
• Synchronization is a Two-Way Street: The system must handle changes originating from the local client and changes originating from the cloud (and other clients). This is typically handled by a combination of local file system watchers and server polling.

Next Lesson Preview:

Our current design has a major flaw: it can't handle large files efficiently. In the next lesson, we will begin our first deep dive. We will focus on the client-side logic for file chunking and synchronization. We'll explore how to break large files into smaller pieces to enable resumable uploads, parallelization, and efficient updates.