RESEARCH METHODS 3RM - Measurement

Author:   Ed Nelson
Department of Sociology M/S SS97
California State University, Fresno
Fresno, CA 93740
Email:  ednelson@csufresno.edu

Note to the Instructor: This is the third in a series of 13 exercises that were written for an introductory research methods class.  The first exercise focuses on the research design which is your plan of action that explains how you will try to answer your research questions.  Exercises two through four focus on sampling, measurement, and data collection.  The fifth exercise discusses hypotheses and hypothesis testing.  The last eight exercises focus on data analysis.  In these exercises we’re going to analyze data from one of the Monitoring the Future Surveys (i.e., the 2017 survey of high school seniors in the United States).  This data set is part of the collection at the Inter-university Consortium for Political and Social Research at the University of Michigan.  The data are freely available to the public and you do not have to be a member of the Consortium to use the data.  We’re going to use SDA (Survey Documentation and Analysis) to analyze the data which is an online statistical package written by the Survey Methods Program at UC Berkeley and is available without cost wherever one has an internet connection.  A weight variable is automatically applied to the data set so it better represents the population from which the sample was selected.  You have permission to use this exercise and to revise it to fit your needs.  Please send a copy of any revision to the author so I can see how people are using the exercises. Included with this exercise (as separate files) are more detailed notes to the instructors and the exercise itself.  Please contact the author for additional information.

This page in MS Word (.docx) format is attached.

Goal of Exercise

The goal of this exercise is to provide an introduction to measurement which is an integral part of any research design.  The other elements of your research design are sampling, data collection, and data analysis and will be discussed in other exercises. 

Part I—Concepts

We use concepts all the time.  We all know what a book is.  But when we use the word “book” we may not be talking about a particular book we’re reading. We could be talking about books in general.  In other words, we’re talking about the concept to which we have given the name “book.”  There are many different types of books – paperback, hardback, small, large, short, long, and so on.  But they all have one thing in common – they all belong to the category “book.”

Let’s look at another example.  Religiosity is a concept which refers to the degree of attachment that individuals have to their religious preference.  It’s different than religious preference which refers to the religion with which they identify.  Some people say they are Lutheran; others say they are Roman Catholic; still others say they are Muslim; and others say they have no religious preference.   Religiosity and religious preference are both concepts.

In other words, a concept is an abstract idea.  There are the abstract ideas of book, religiosity, religious preference, and many others.  Since concepts are abstract ideas and not directly observable, we must select measures or indicants of these concepts.  We’ll call this process measurement.

Assume that we’re interested in the following research question:  Why do some people favor same-sex marriage and others oppose it?  In other words, we’re trying to explain support or opposition to same-sex marriage.  Let’s think about the concepts that might help us explain such support.  If we think that females are more likely to support same-sex marriage then men, then sex would be a concept that we would want to include in our study.  If we think that people with more education are more likely to support same-sex marriage, then education would be a concept to include.  But not all concepts would be relevant to our study of same-sex marriage.  Hair and eye color are concepts but they wouldn’t be relevant. 

Part II – Now It’s Your Turn

The research study that we’ll be using in these exercises is the Monitoring the Future Survey of high school seniors in the United States that has been conducted yearly since 1975.  A major focus of this survey is students’ drug use.  Questions about drug use include a variety of questions about alcohol.  One of these questions asked how often during the last two weeks students had consumed “five or more drinks in a row” which is a common definition of binge drinking.  Let’s assume that we want to explain why some students engage in binge drinking and others do not.  List three concepts that you think might help us understand this behavior.  For each concept write a paragraph indicating why you think this concept would be useful in explaining binge drinking.

Part III – Measures

A concept is an abstract idea.  Abstract ideas can’t be directly observed.  We have to find some piece of empirical data that we can use as a measure (or indicant) of the concept.  Let’s see if we can make this clearer by looking at some examples. 

Let’s start by looking at how the Monitoring the Future Survey measured students’ drug use.  Our concept is drug use.  First, the survey recognized that there are many different types of drugs that students might use (e.g., alcohol, marijuana, cocaine, LSD, etc.).  Use of each of these drugs needs to be measured separately.  For each drug the survey asked how often students had used this drug over their lifetime, during the last twelve months, and during the last 30 days.  In other words, students’ answers to these questions become our measures of this particular type of drug use.

Now let’s consider political views or outlook.  This refers to whether respondents are politically conservative, moderate, or liberal.  How could we measure this concept?

One way to measure political views is to ask respondents the following question – “In general, would you describe your political views as conservative, moderate, or liberal?”  In other words, respondents are asked to self-report their own political views.[1]

Another way is to use a series of questions that asks respondents whether they thought government was wasteful, whether government regulation was necessary, whether the government should do more to help needy Americans, and whether the best way to ensure peace was through military strength.[2]  Each of these questions had a clear conservative response and a clear liberal response.  You could count the number of conservative and the number of liberal responses and use this to measure political views. 

Finally, let’s think about how we could measure voter turnout.  The Pew Research Center has shown that voter turnout in U.S. presidential elections is relatively low (i.e., relative to other developed countries).[3]  Voter turnout is the concept.  But how do we measure voter turnout?  One alternative would be to divide the number of votes cast by an estimate of the voting-age population.  Another alternative is to divide the number of votes cast by the number of registered voters.  In 2016, 55.7% of the voting age population voted but 86.8% of registered voters actually voted.  What accounts for this difference is that in the United States a lot of people who are eligible to vote don’t actually register and therefore cannot vote.  The way in which we measure concepts often makes a difference so it’s critical to be clear in explaining how we measure our concepts.

Part IV – It’s Your Turn Again

Here are three concepts that you might want to include in any study that focuses on political behavior.  Write a paragraph for each concept indicating how you might measure that concept.  Include the questions that you would want to ask in your survey.

• Political party preference – the political party that the person identifies with.  Be sure to allow for the possibility that a person might identify with a party other than the Republican and the Democratic parties and the possibility that the person has no political party preference.
• Voting behavior – the candidate that the person is most likely to vote for if the election were held today.  You can choose any future political election.
• Likelihood of voting – the likelihood that the person will vote in an upcoming election.

Part V – Characteristics of Good Measures

You might be wondering how we distinguish between good and bad measures.  Good measures are reliable, valid, and precise enough.[4]

Reliability means that they are consistent.  Consistency refers to two different aspects of measures. 

• Imagine that you weighed yourself in the morning.  Then you took a shower and when you got out of the shower you weighed yourself again.  You would expect the scale to give you very similar results.  If you weighted much less the second time you weighted yourself, you would question the reliability of your scale.  This refers to consistency over time.  The way you check for consistency over time is test and then retest.  For example, if you wanted to check on the reliability of your question about same-sex marriage, ask a sample of respondents at time one and then retest by asking the same respondents the same question again at time 2 (e.g., one month later).  A few respondents might change their answer but most should give you the same response.
• Most of us have taken the written test for our driver’s license.  It’s a series of multiple-choice questions.  You have to get a certain number of questions correct to pass.  There isn’t just one test.  If there was, everyone would know the questions before they took the test and probably pass.  There’s a whole set of tests with different questions.  However, we wouldn’t want some of the tests to be harder than others.  We want the tests to be equivalent.  We refer to this type of consistency as equivalence. 

We also want our measure to be valid.  A measure is valid if it measures what we say it measures.  We can test for validity in several different ways.  Here are some of these ways.

• Face validity means that the measure is valid on the face of it.  There are some measures that are obviously valid.  For example, we might ask how old a person was on their last birthday.  We would probably all agree that this measure of age is obviously valid.  Notice by the way that there are some situations where we wouldn’t consider it obviously valid.  A bartender wouldn’t accept a patron’s answer to this question as obviously valid.  The bartender would want to see a photo identification such as a driver’s license.
• When you apply for a credit card, a company must have some way to decide whether to give you that credit card.  But how do they measure your credit risk?  Are you a good risk or a poor risk?  Let’s say they select a sample of 1,000 clients from all their past clients.  Then they randomly divide this into two subsamples of 500 clients each. They use the data from the first subsample to develop their credit risk measure.  This measure scores applicants on a measure of credit risk that varies from 1 (very poor risk) to 10 (very good risk).  Then they use that measure to see if it accurately predicts whether the second random subsample paid their bills or not.  If there is a strong correlation between the credit risk measure and their credit history, then they have validated their measure.  We call this predictive validity because we can determine whether our measure accurately predicts this clearly valid measure of one’s credit risk.  If it is valid, then they can use this measure to make future decisions about who should be given credit cards because they know it is a valid measure of credit risk.
• Let’s say that we want to measure a person’s mathematical skill.  Mathematics encompasses many content areas – simple arithmetic, algebra, geometry, trigonometry, calculus, and beyond.  So we decide to develop a test to measure mathematical skill.  Our test should include all these areas and not just some of them.  If it does, we can say that it has content validity since it covers the entire range of mathematical skills.

We also want our measure to be precise enough.  If you want to know what the temperature currently is, you probably only need to know it to the nearest degree.  It’s sufficient to know that it is 84 degrees Fahrenheit.  You don’t need to know that it is 84.2 degrees.  Similarly, if you want to know a person’s family income, you probably only need to know it to the nearest thousand dollars.  More precision would probably be unnecessary.  On the other hand, you might want more precision than just knowing income to the nearest tens of thousands of dollars.  You want your measure to be precise enough.