Understanding Data Formats: Structured, Semi-structured, and Unstructured

The Poles of Data Organization: Structured and Unstructured

Let's begin by examining the two extremes of data organization. At one end, we have perfectly ordered data, and at the other, we have data with no inherent order at all.

Structured Data: The Neatly Organized Spreadsheet

Structured data is highly organized and follows a rigid, predefined model. Think of a well-organized spreadsheet or a traditional database table. It's composed of rows and columns, where each column is for a specific type of data (like a name, date, or number) and each row represents a single record.

Because of this rigid structure, it's easy for both humans and computers to search, query, and analyze. Given your pharmacy background, you've likely encountered structured data in many forms. A pharmacy's inventory system, for instance, would store drug information in a structured way:

The following article from Datapharm, a company focused on medicines information, explains this concept using a very relevant example: the Summary of Product Characteristics (SmPC).

Unstructured Data: The World of Free-Form Information

On the opposite end of the spectrum is unstructured data. This is information that does not have a predefined data model or organization. It's essentially the vast majority—an estimated 80-90%—of the data generated today.

Common examples include:

• Text documents, emails, and social media posts
• Images, videos, and audio files
• Scanned documents, like a handwritten prescription or a signed consent form

In a healthcare setting, unstructured data is incredibly valuable but challenging to analyze. Think of a doctor's detailed clinical notes, a radiologist's report interpreting an X-ray, or a patient's call to a pharmacy describing a potential side effect. This information is rich with context but doesn't fit neatly into a table.

Let's use a short video from business author Bernard Marr to get a clear, high-level definition.

The Middle Ground: Semi-Structured Data

Between these two poles lies semi-structured data. As the name implies, it's a hybrid. It doesn't conform to the rigid structure of a relational database, but it does contain tags, markers, or other forms of metadata to create a hierarchy and separate semantic elements. This makes it more organized than unstructured data and easier to process.

Common formats for semi-structured data include XML (eXtensible Markup Language) and JSON (JavaScript Object Notation), which are widely used for data exchange between different systems and applications.

Bernard Marr, in the same video, provides an excellent example using an email. The body of the email is unstructured text, but it is wrapped in structured metadata: sender, recipient, subject, and timestamp.

In healthcare, a critical example is the Health Level Seven (HL7) standard, used to exchange clinical and administrative data between different healthcare providers' software systems. Another great example, provided by Datapharm, is an electronic Patient Information Leaflet (ePIL).

Comparing the Three Formats

Now that we've defined each type, let's watch a video that provides a fantastic summary and comparison, touching on their characteristics, storage methods, and pros and cons. This will help solidify your understanding and connect back to our previous lesson.

To bring everything together, the following table provides a side-by-side comparison of the key attributes of each data format.

As you can see, the format of the data has profound implications. Structured data is easy to analyze with tools like SQL, which we'll learn about later. Unstructured data requires advanced techniques like Natural Language Processing (NLP) to extract meaning from text or image recognition for analyzing medical scans. Semi-structured data falls in between, often requiring a parsing step before analysis can begin.

Conclusion

In this lesson, we have broken down the different ways data can be organized, completing our foundational understanding of data itself.

Key Takeaways:

• Structured Data is highly organized, has a fixed schema, and fits neatly into tables (rows and columns). It's easy to analyze with traditional tools. Example: A patient database with columns for name, age, and blood type.
• Unstructured Data has no predefined model or organization. It includes things like text, images, and audio. It is rich in information but difficult to analyze. Example: A doctor's handwritten notes.
• Semi-Structured Data is a hybrid. It doesn't have a rigid tabular structure but uses tags or other markers to create a hierarchy. Example: A JSON file of patient lab results or an HL7 message.
• The data format dictates the storage methods, tools, and analytical techniques you can use.

You have now completed the first module, "Foundations of Data Analytics." You can define data analytics, describe its key process and types, and now differentiate between data's fundamental types (quantitative/qualitative) and formats (structured/semi-structured/unstructured).

With this foundation in place, we are ready to get our hands dirty. In our next lesson, we will begin working with the most common form of structured data. You will learn to navigate spreadsheet interfaces and organize data using rows, columns, and sheets, taking your first practical step into the world of data analysis.