Critiquing Research Questions with SMART Criteria

A Framework for Critique

Before we dive into examples, let's establish a clear process for critiquing a research question. A systematic approach ensures that no criterion is overlooked. The following resource from the CDC provides a helpful checklist format that we can adapt for our purpose.

Drawing from this resource, our critique process will be to ask a series of questions about the research proposal, one for each letter of the SMART acronym:

1. Specific: Is the target population ("who") and the activity/variable ("what") clearly defined?
2. Measurable: Is the outcome quantified? Is it clear "how much" change is expected or how the variable will be measured?
3. Achievable: Can this be realistically done given typical constraints on time, budget, and access to participants?
4. Relevant: Does this question clearly connect to a broader problem or a larger research goal?
5. Time-Bound: Is there a clear timeline for when the data will be collected or when the question will be answered?

A "no" to any of these questions signals a weakness in the research question that needs to be addressed.

Application 1: Critiquing a Bra Design Question

Let's start with a common type of question that might emerge in the early stages of product development.

Poorly Formulated Question: "To determine if softer materials make bras more comfortable."

At first glance, this seems like a reasonable goal. But let's apply our SMART critique.

• Is it Specific? No.

  • "Softer materials" is vague. Are we talking about modal, microfiber, or a new proprietary blend? It needs to be defined.
  • "Bras" is too general. A specific style or construction (e.g., a non-wired bralette, a full-coverage underwire bra) should be identified, as the interaction between material and structure is critical.
  • "More comfortable" than what? A specific baseline material (like cotton or nylon) is needed for comparison.
  • For whom? Comfort is subjective and varies greatly across body types and breast shapes. The target wearer (e.g., women in cup sizes DD-G) must be specified.

• Is it Measurable? No.

  • "Comfort" is not an inherent metric. How will it be measured? Through a self-reported 1-to-10 scale? By tracking the duration a participant can wear the bra before reporting discomfort? By counting the number of physical adjustments a wearer makes during a day? The question must imply a metric.

• Is it Achievable? No.

  • As stated, the question implies testing all softer materials against all other materials, which is impossible. A study comparing a small, defined set of materials is achievable, but this broader question is not.

• Is it Relevant? Unclear.

  • The relevance is only implied. Is the goal to reduce product return rates? To support a new marketing campaign focused on comfort? To solve a known issue with a current product line? A well-formulated question is relevant to a specific, stated objective.

• Is it Time-Bound? No.

  • There is no timeframe for the study. Is this a one-day wear test or a multi-week trial?

Critique Summary: This question fails on all five SMART criteria. It is too vague and lacks the necessary detail to be a workable research plan. It uses subjective words without defining how they will be operationalized.

The following resource provides excellent guidance on why avoiding vague and subjective language is so important.

Application 2: Critiquing a Relationship Dynamics Question

Now let's apply the same rigorous critique to a question from the domain of social research.

Poorly Formulated Question: "Investigate how polyamorous people manage their schedules."

Again, this is a topic of genuine interest, but as a research question, it is not ready for analysis.

• Is it Specific? No.

  • As the image above illustrates, "polyamorous people" is a very broad category. The dynamics of a Vee relationship are vastly different from those of a Quad or a Solo Polyamorist. The question must specify the relationship structure being studied (e.g., one-female-multiple-male triads).
  • "Manage their schedules" is also vague. What aspects of scheduling are we interested in? Allocating time for one-on-one dates? Coordinating domestic responsibilities? Negotiating holidays with multiple partners and metamours?

• Is it Measurable? No.

  • The word "how" points toward a descriptive goal, but the verb "manage" does not provide a measurable outcome. Are we trying to quantify the time spent on scheduling tasks, the satisfaction with the resulting schedule, or the frequency of scheduling conflicts? As a statistician, you know that without a clear variable to measure, no analysis is possible.

• Is it Achievable? No.

  • Surveying or observing a representative sample of "all polyamorous people" is not feasible due to the size and diversity of the population and the challenges in recruitment. Focusing on a well-defined subgroup is necessary.

• Is it Relevant? Unclear.

  • What is the purpose of this investigation? Is it to develop a best-practice guide for polyamorous individuals? To create a scheduling app tailored to their needs? To contribute to sociological theories about household management? The question lacks an explicit link to a larger objective.

• Is it Time-Bound? No.

  • The question provides no timeframe. Should the data be collected over a week, a month, or a year? The lack of a time boundary makes it an indefinite exploration rather than a focused study.

Critique Summary: This question also fails on all five SMART criteria. Its terms are too general, it lacks a measurable outcome, and its scope is unfeasibly broad.

Your Turn to Critique

Now, apply the SMART framework to critique the following two questions. For each one, identify all the criteria it violates and briefly state why.

Question 1 (Bra Design): "To find out if wire-free bras are comfortable."

Question 2 (Relationship Dynamics): "To understand jealousy in polycules."

Take a few minutes to analyze them before reading the breakdown below.

Analysis of Question 1: "To find out if wire-free bras are comfortable."

• Specific: Fails. Which wire-free bras? For whom (what body type/size)?
• Measurable: Fails. How is "comfortable" being measured?
• Achievable: Fails. Testing all wire-free bras is impossible.
• Relevant: Fails. "To find out" is not a sufficient goal. Relevant to what decision or problem?
• Time-Bound: Fails. No timeframe is given.

Analysis of Question 2: "To understand jealousy in polycules."

• Specific: Fails. What kind of jealousy (e.g., sexual, emotional, time-based)? In which members of the polycule (e.g., new vs. established partners)? What is the specific polycule structure?
• Measurable: Fails. "Understand" is not a measurable outcome. What would be measured? The frequency of jealous feelings? The intensity? The coping strategies used?
• Achievable: Fails. "Polycules" is too broad and varied.
• Relevant: Fails. "To understand" is a starting point, but why is this understanding needed? For therapeutic intervention? For community education?
• Time-Bound: Fails. No timeframe is defined for the study.

Conclusion

In this lesson, we synthesized our knowledge of the SMART framework to use it as a powerful tool for critique. We have seen how seemingly reasonable questions about bra comfort and relationship management fall apart under systematic scrutiny.

Key Takeaways:

• Critiquing a research question is a systematic process of checking it against all five SMART criteria: Specific, Measurable, Achievable, Relevant, and Time-Bound.
• Common failures include using vague terms (e.g., "better," "happiness," "manage"), failing to define the population, and lacking a quantifiable metric.
• This critical process is not about dismissing ideas but about identifying weaknesses so they can be strengthened. It is the essential bridge between a vague interest and a viable research project.

Preview of the Next Lesson:

Now that you have mastered the art of critique, the next logical step is reformulation. In our final lesson of this module, you will take a poorly formulated question, like the ones we analyzed today, and rewrite it into a fully compliant SMART research question. This will complete the journey from a raw idea to a polished, actionable plan for data collection and analysis.