File Systems: Limitations for Structured Data

The World Before Databases: The File System Approach

Imagine you're building an application for a small college. You might decide to store student information in one file (e.g., students.csv), course details in another (courses.txt), and enrollment records in a third (enrollments.dat). Each file would be managed by a different part of your application, perhaps even written in different programming languages like C or Python.

This is the traditional "file system approach." Each application directly accesses and manipulates its own files.

While this seems simple enough for small tasks, it quickly breaks down as the data and the number of users grow. Let's start by exploring the most common and immediate problems this approach creates.

1. Data Redundancy and Inconsistency

When data is scattered across different files, you inevitably end up with duplication. This is called data redundancy. For example, a student's name and address might exist in the registrar's file, the library's file, and the billing department's file.

What's wrong with that?

• Wasted space: Storing the same data multiple times is inefficient.
• Data inconsistency: This is the more serious problem. If a student moves and updates their address with the registrar, what ensures the library's and billing's files are also updated? If they are not, the data becomes inconsistent. The system now holds conflicting information, and you don't know which version is correct.

To see a clear explanation of these two interconnected problems, please watch the following video segment.

This issue of redundancy leading to inconsistency is a core reason why managing data in separate files is so fragile. In your work with REST APIs, you can think of it like having the same configuration parameter hard-coded in multiple microservices. Changing it in one place but forgetting another leads to system-wide inconsistencies that are difficult to debug.

2. Difficulty in Accessing Data and Data Isolation

What if the college dean wants a list of all students from a specific city who are enrolled in an advanced mathematics course?

With data stored in separate files (e.g., a students file and an enrollments file), there's no straightforward way to answer this question. You would need to:

1. Write a custom program or script.
2. This script would have to open both files.
3. It would need to parse them, understand their different formats, and then manually link the data (e.g., by matching a student ID).
4. Finally, it would filter the results based on the dean's criteria.

This process is slow, inefficient, and requires new programming effort for every new type of query. This is the "difficulty in accessing data" problem.

Compounding this is data isolation. Because the data is scattered across different files, possibly in different formats, it's difficult to get a single, unified view of the information.

3. Integrity Problems

Data integrity refers to the correctness, consistency, and completeness of data. File systems offer no built-in way to enforce rules (or constraints) on the data they store.

For example, how would you enforce these business rules in a simple text file?

• An employee's salary must be a positive number.
• A student's GPA must be between 0.0 and 4.0.
• Every course must have a unique course ID.

You could write checks for these rules into every application program that modifies the file, but that's unreliable. If a new application is written, or an existing one is updated carelessly, invalid data can easily be introduced.

This lack of integrity enforcement leads to data anomalies.

4. The "Hard" Problems: Atomicity, Concurrency, and Security

The final set of limitations are the most complex and are particularly relevant to your interests in systems and low-level programming. These are the problems that are exceptionally difficult to solve correctly using a file-based approach.

• Atomicity Problems: Many real-world operations consist of multiple steps. A classic example is a bank transfer: (1) debit account A, (2) credit account B. What happens if the system crashes after step 1 but before step 2? The money is gone from A but never arrived at B, leaving the data in an inconsistent state. An operation that is atomic is "all or nothing"—it either completes fully, or it doesn't happen at all, leaving the system unchanged. File systems have no mechanism to guarantee this.

• Concurrent Access Anomalies: What happens when two users try to modify the same file at the same time? For example, two airline agents trying to book the last seat on a flight. Both might read that "1 seat is available" and proceed to sell it. This is a classic race condition. Simple file systems often handle this crudely by locking the entire file, preventing the second user from even reading it, which is highly inefficient.

• Security Problems: How do you enforce granular access control? For instance, in an employee file, you might want managers to see the salary column but prevent other employees from seeing it. A file system typically only offers all-or-nothing protection at the file level (e.g., via OS permissions). It's very difficult to control access to specific parts of a file for different users.

These three challenges—ensuring atomicity, managing concurrency, and providing fine-grained security—are where file systems truly fall short for modern applications.

Conclusion

As we've seen, while storing data in files is simple on the surface, it introduces a host of serious problems as soon as your data has any structure or is accessed by more than one user or application.

Key Takeaways:

• Data Redundancy & Inconsistency: Duplicating data wastes space and, more importantly, leads to conflicting versions of the truth.
• Difficult Data Access & Isolation: Querying data is cumbersome, requiring custom programs, and getting a unified view of scattered data is challenging.
• Integrity Problems: File systems cannot enforce business rules, leading to invalid data and anomalies.
• Atomicity Failures: "All or nothing" operations are not guaranteed, making the system unreliable in case of failures.
• Concurrency Issues: Simultaneous access by multiple users can corrupt data or lead to incorrect results.
• Limited Security: Providing granular, data-level security is nearly impossible.

These limitations are precisely the problems that Database Management Systems (DBMS) were designed to solve. A DBMS isn't just a fancy file holder; it's a sophisticated software layer that sits between your applications and the data, providing solutions for every one of these issues.

Preview of the Next Lesson:

Now that we have a firm grasp of the problem, our next lesson will introduce the solution. We will begin to explore what a Database Management System (DBMS) is and describe how its core components are specifically architected to tackle challenges like data integrity, concurrent access, and recovery from failure.