Statistics Consulting Blog

Building the Research House

2011-02-24T12:36:00.005-05:00

Imagine that you have been given a huge toychest filled with toy parts and pieces. Your goal is to build a toy house from these mix of pieces. As you will see, this is not all that different from certain aspects of the research process.

In much of research, our goal is to predict or explain one variable from a set of other variables using the multiple regression technique. The variable that you predict or explain is usually called the dependent variable (or response variable). The variables that do the predicting or explaining are usually called the independent variables (or predictor variables). So, in our example, the house is the dependent variable, and all of those pieces are the independent variables.

We may know ahead of time that certain pieces fit well with our conception of a house. For example, if our goal is a log-cabin, any toy piece made of wood might get automatically included in the mix. What about a toy necklace? Well, the beads might be used for decorations. But is it necessary to include them? On the other hand, why not just include every toy piece?

In multiple regression, we are faced with similar decisions to make. Suppose we are trying to predict student math test scores. An obvious predictor would be exposure to training (i.e., attending a math course with content related to that math test), which can be measured by attendance. We might also include time spent studying in hours. We could go a step further and include variables such as presence of a tutor/mentor and family socioeconomic status. Technically, we could go even further...adding every toy piece from the chest...because, in reality, almost everything is related to everything in some way, shape, or form. So, how do we handle this?

1. Keep the model as simple as possible. Do not include extraneous variables just for the sake of including them. Have a strong rationale for including every variable that you include in the model.

2. Keep the model as complex as necessary. In contrast, to the previous advice, do not omit variables simply because you decided to keep it simple. If you liked having three predictors, but ignoring a fourth predictor would lead to an incomplete "house", by all means...include the fourth predictor. What you want is a set of variables that represent the majority of what you are predicting; variables that have negligible impact can be omitted from the research.

3. Ensure that your proxy variables resemble reality as much as possible. For example, in educational research, we often measure socioeconomic status using the percent on free and/or reduced lunch programs as a proxy. And, this is currently argued to be the best proxy available.

4. Base all decisions on strong theory. Attempt to find literature to justify your choice to include or ignore variables.

5. Document anything that might help future researchers, and include this in your discussion. Did you ignore a particular variable because of lack of theory? Future researchers could develop a theory. Did you ignore a particular variable because of lack a decent proxy or available data? Future researchers might tackle that problem (or not be dealing with it).

6. Acknowledge that human behavior includes inherent randomness that will not be explained by any particular set of variables.

So, returning to the toychest, what we want is basic but requires much planning. We want a house that uses the minimum number of toy parts. However, we want as many toy parts as necessary to ensure that we have built something resembling a house. We connect our choices to preconceived notions such as wanting a particular style of house. We tell others that we didn't include certain pieces, and we tell them why since they might want to attempt this in the future. Finally, we accept that our final product will not be a perfect toy house; however, we made every attempt to create as perfect of one as possible.

For consultations, please send an e-mail to
Dr. Miller

The Biggest Loser (Regression Style)

2011-02-21T13:04:00.004-05:00

Our goal with many research questions is to determine which variable is the biggest predictor of something. For example, we may want to predict endorsement for a public policy from variables such as income, gender, and party loyalty. Which predictor is strongest? Alternatively, which is the biggest loser? Is it the respondent's income? Is it gender? Or, is it some unobservable such as the degree to which they are loyal to a particular party?

The statistical technique we use here, multiple regression, is fairly elementary. However, the interpretation of results is often erroneous. This is because we often want to compare the regression coefficients in the output to decide which predictor is strongest.

Why might this be erroneous? Consider a simple example of comparing height and temperature. Is a height of 40 inches greater than a temperature of 24 degrees? We can't even answer that question. It's a comparison of apples and oranges. Technically, they are on different scales.

In statistics, we typically standardize variables to put them on a level playing field so as to more validly make comparisons. In the original example, we shouldn't compare something like income in dollars to party loyalty, which might be on a 10-point agreement scale. So, we would want to look at the standardized regression coefficients, not the unstandardized regression coefficients. (In common language, these standardized regression coefficients are sometimes loosely called "the betas"; however, we should seek clarification because, technically, the unstandardized regression coefficients use the Greek beta symbol.)

The downfall to standardized regression coefficients is that they are more difficult to interpret. Since the standardization puts the variables on standard deviation scales, we wind up interpreting in terms of changes in standard deviations. So, we would still want to look at unstandardized regression coefficients when wanting to explain how much the response variable changes on average given a one-unit change in the predictor variable. The standardized regression coefficients are the better choice when wanting to determine which variable is the strongest predictor of the response variable...the biggest winner...or the biggest loser.
For consultations, please send an e-mail to
Dr. Miller

The Delphi Technique

2011-02-18T17:44:00.008-05:00

One of my first statistics consultations was for a graduate student using the Delphi Technique. I had never heard of it but was assured that the statistical part of it would not be difficult. Part of me was fascinated by learning a new technique; the other part was hoping that I might gain some prophetic insights as if making a pilgrimage to the ancient Oracle of Delphi in Greece.

What is the Delphi Technique? At first glance, it seems like a qualitative technique. You sit down with a group of experts to discuss something. That's a qualitative focus group method, right? Ah...but we take things a step further.

Suppose that I want to know how arts educators feel about standardized testing within their field. I might assume they would be against it since such testing in other areas is perceived to distract from their area. On the other hand, they might be for it since it would heighten validity for their field by being included alongside other subjects such as math and writing.

I start the technique by having experts reply to open-ended questions such as, "How do you feel about including music on your state's standardized annual assessment? Describe in detail." Now, that is definitely qualitative! But, then I take the responses and use them to develop a survey with response options that are quantitative (e.g., a Likert scale). Alternatively, I might identify shared themes and then construct the survey such that themes can be rank-ordered in terms of things like favorability and priority.

Next, I move on to what we would call Round 2. The same experts respond to the survey developed from the open-ended questionnaire. Now, this is still looking like some mixed-methods twist on a focus group technique. What sets this method apart from others is Round 3.

In Round 3, I actually share the results with the experts. In other words, Expert A gets to see the responses from Expert B, C, and D. They then take the survey again thereby having the opportunity to adjust their previous responses. This step is usually repeated once more (i.e., Round 4); however, the process could continue through several iterations.

Statistical analysis enters here when trying to determine an optimal level of agreement or merely wanting to display the results. I could use means and standard deviations for this. However, given the relatively small sample size used here, I might want to use the more sophisticated Score interval adapted by Randall Penfield at the University of Miami and myself for this process (Miller & Penfield, 2005).

The Delphi technique is an excellent technique for studies involving expert agreement and/or consensus building. It combines a traditional qualitative approach with the potential for rigorous quantitative analysis. Finally, it answers research questions that often have practical implications.

The Delphi Technique was created by the Rand Corporation.
For consultations, please send an e-mail to
Dr. Miller

What "Causes" What??

2011-02-17T17:40:00.005-05:00

I was thinking today about how loosely we use the term "cause" in research. We go into research wanting to demonstrate (or explore the possibility) that something causes something. We may be interested in finding out if having loving parents as a child causes us to be giving adults. We may want to show that video games cause violent behavior...or maybe that being predisposed toward violence causes one to be interested in playing video games.

The truth of the matter is that we can not determine cause from statistical results. We may get that wonderfully low "p-value" for statistical significance...but even that says little to nothing about causation. Our results may replicate previous findings; yet, that still is not sufficient for causation. It may make intuitive sense, and everyone might nod their heads in agreement...but 400 friends agreeing that eating eggs causes one to spend more time on the Internet does not make that true....even if the results of a statistical analysis were to suggest that :)

Determining causation is actually a matter of philosophy and depends on things like the design of the experiment, itself....not the statistical analysis. In fact, one could earn a doctoral degree in philosophy with a dissertation on nothing but the topic of causation. Depending on the source and theoretical framework, there are many sets of criteria one could choose from for defending causation. For the purposes of research results interpretation, we are better suited to use phrases such as

"the results of the analysis suggest that _____ may cause _____"

rather than

"the results prove that _____ causes _____ .

For consultations, please send an e-mail to
Dr. Miller

Dissertation Editing

2009-05-11T18:27:00.006-04:00

Writing a dissertation is a painstaking process. It usually begins with ideas floating around in the mind until it comes time to materialize it on paper. A few sentences hopefully inspires a thorough literature review. The wise student is meticulous in organizing specific ideas from hundreds of articles such that they tie into a cohesive and well-organized chapter of the dissertation proposal.

During this process, the information load usually exceeds cognitive capacity to synthesize a perfect draft. More than often, the chapter is an amalgamation of statements, often disjointed with paragraphs related to a theme disconnected throughout the manuscript. Revisions are in order. The goal here is to get that proposal accepted so that the student can move on to the actual research! So, it is vital that the literature review and the preceding introduction chapter be clear in flowing from the statement of the research problem to its justification and substantiation via previous research.

There are two stages that are also tricky in terms of editing once the research is complete. First, there are the nuances of APA-format required for Chapter 4. For example, most of us don't naturally think to write "was statistically significant, t(38) = 2.19, p = 0.035" with full knowledge of what to italicize and where to place spaces. Second, somehow that results chapter and the following discussion chapter have to connect with the preceding chapters in such a way that the entire dissertation tells a complete and coherent story.

Please send requests for more information to jeffmiller.research@gmail.com

2009-03-24T22:02:00.005-04:00

My Table Has Too Many Zeros

2008-09-01T23:39:00.003-04:00

Hi everyone,

Sometimes we are interested in looking at the association between two qualitative variables such as gender and political party. We ask, "Is there a significant association between gender and political party?", and we typically use a chi-square test of independence in the statistical analysis.

We usually assume an expected count of 5 in each cell but this doesn't always happen. SPSS will still provide output but include a note stating something like "50% of the cells have an expected count less than 5". What to do?

Well, one solution is the Fisher's Exact Test. This is a nonparametric test that utilizes the hypergeometric distribution.

Some software programs state that it only works for a 2 X 2 table such as Gender (male and female) by Political Party (Democrat and Republican). This simply isn't true. It's just very computer intensive.

Please send requests for more information to jeffmiller.research@gmail.com

Best regards,
Jeff

Being Wrong "On Average"

2008-06-27T09:34:00.003-04:00

One of the most simple concepts we learn in a statistics course is that of the "average". Most of us come into a statistics course already familiar with the concept of adding up a bunch of numbers and then dividing that total by the number of things being added. For example, if three people are of ages 10, 20, and 30 then the average is simply (10 + 20 + 30) / 3 = 20. We learn that, in statistics, this average is called the "mean". Sometimes, we learn the more complete word for it -- the "arithmetic mean".

Unfortunately, we often report the wrong average. Believe it or not, there is actually more than one way to calculate the "mean", and the best one to use depends on what we are doing with the numbers. For example, suppose you are told that you will receive salary increases each year as follows.

* Year 2 - 2.0%
* Year 3 - 2.5%
* Year 4 - 2.9%
* Year 5 - 3.5%
* Year 6 - 4.2%

If your starting pay was $40,000 USD then, to get your Year 2 pay, you would multiply 40,000 X 1.02 = 40,800. To get the Year 3 pay, you would then multiply that 40,800 X 1.025 = 41,820. Notice that in each case we are "multiplying", not adding. Hence, to get the average percent pay increase, we should calculate a mean based on multiplying. The arithmetic mean would not be the most correct average to report.

Fortunately, we can use the geometric mean to find the answer to this problem. First, we multiply, not add, all the numbers together. Let k = the number of items being multiplied. Now, just take the k-th root.

Note that if you only multiplied 2 numbers, you would take the 2nd root, which is the square root. The 3rd root is often called the cubed root. In our example, we are multiplying 5 numbers, so we take the fifth root.

Here's how we calculate the answer in Excel.

1.) Convert the percentages to the numbers we would use in multiplication.

* Year 2 - 1.02
* Year 3 - 1.025
* Year 4 - 1.029
* Year 5 - 1.035
* Year 6 - 1.042

2.) Enter these values in Excel. I put them in cells A1 through A5.

3.) In cell A6, I multiply by typing =PRODUCT(A1:A5) which gives me a multiplicative solution of 1.16

4.) Take the 5th root by raising the product to the 1/5th power. In cell A7, I type =A6^(1/5) giving a solution of 1.03

An easier method in Excel would be to just type =GEOMEAN(A1:A5) and skip steps 3 and 4. It grants the same solution.

So, the average pay increase over these 5 years is 1.03, or 3%. Note that the arithmetic mean would also suggest an "arithmetic" average increase of 3% but this isn't always the case. Further, I rounded to two decimal places. If we were dealing with large numbers and more decimal places then the results might be more meaningfully different.

Try your hand at it. A stock has yielded the following changes over the past 5 quarters: 4%, 7%, 12%, 10%, and 11%. What is the average increase? How do your results differ when using the typical arithmetic mean versus the more appropriate geometric mean?

Please send requests for more information to jeffmiller.research@gmail.com

Stating Testable Hypotheses

2008-01-05T11:55:00.001-05:00

Provided that one has a properly framed research question, the next step is to rephrase it in a manner that can generate an answer. The question is permitted to be vague; however, the hypotheses must be concrete. We must bring the question down from the realm of ideas and settle it on solid earth.

The research question communicates your "Hmmm, I wonder..." as "Hmmmm, I think..." So, in that sense, the hypothesis does remain a bit vague but only because aren't making any judgments of knowing. It is the words that we choose in our hypothesis that must be concrete.

Here is an example of the research question:

Research Question: Are children are more resilient than we think they are?

See how vague that is? What do we mean by resilient? How would we know if they are more resilient than 'we think' they are?

Now, here's a hypothesis

Hypothesis: Childrens' scores on the Resilience Scale will negatively correlate with their parents' scores on the Perceptions of Child Resilience Scale.

We have declared a way to measure resilience and a way to measure parents' perceptions of childrens' resilience via the two Scales (i.e., surveys). We have also declared a way to compare the two via "negatively correlate"

This hypothesis would suggest that lower scores for parents' perception are associated with higher scores for childrens' perception. Parents with low belief in their childrens' perception <--> Children with high resilience.

Note that this only an example of how to state a hypothesis. There are many other concerns not addressed here (e.g., what is meant by children?). Note also that this only one of many ways that the hypothesis could have been stated.

Strictly speaking, the hypothesis should be stated to the contrary as a "null hypothesis" symbolized H0:

H0: Childrens' scores on the Resilience Scale will NOT be negatively correlate with their parents' scores on the Perceptions of Child Resilience Scale.

We accept the null as the truth. Our results can then tell us how much we can trust "our" hypothesis, often called the "alternative hypothesis" symbolized H1: or Ha:

Given a stated testable hypothesis, we can then more efficiently gather data, conduct trials, analyze the data, and interpret results.

Please send requests for more information to jeffmiller.research@gmail.com

Framing Research Questions

2008-01-02T08:56:00.001-05:00

Framing the research question is similar in difficulty to writing the introduction paragraph in a paper. It forces you to engage in effortful and deep levels of cognitive processing. It requires you to objectively state what it is that you are researching.

Why is it so difficult to do this and do it well?

It's a cognitive workout. We humans are suited for automating simple tasks in order to pursue more complex goals. For example, the mere act of getting out of bed involves hundreds of decisions that engage hundreds of muscles; yet, we take this for granted and just get out of bed. Similarly, our research topic is probably very familiar to us. We may think, "Well, my question is obvious to me. Why detail it out in just a few words?

Failure to properly frame a research question seriously puts the cart before the horse. It has been said that a thousand-mile journey begins with a single step. But in which direction? Which foot? Which journey?

Thomas Carlyle stated that "a man without a goal is like a ship without a rudder". In the case of framing a research question, you may still have a rudder but you may have forgotten your map. Or you may have a map that is upside-down. Worse yet, you may have brought the wrong map.

TRICKS AND TIPS
1. Read other research papers and see how it was done. Draw from experience.
2. Explain your research to others in their language. Don't just say, "Well, it's a bit complicated."
3. Write several questions.
4. Don't worry when your question generates other questions. This is a good thing! Write those down and come back to them.
5. Write operational definitions for the terms that might appear in your research questions.
6. Make a concept map.

Please send requests for more information to jeffmiller.research@gmail.com

Demystifying the Binomial Distribution Formula

2007-11-23T18:57:00.002-05:00

So, we're taking a stats class and reading the text. We're learning about probability, and all is going well. Of course flipping a coin has a heads probability of .50! Of course the probability of rolling a 6 on a fair 6-sided die is 1/6! Stats is supposed to be difficult?

Then we find out that we were just learning about ONE flip of the coin and ONE roll of the die. Dr. Quantoid tell us, "It is estimated that 20% of the fish in my lake are large-mouth bass. What is the probability of catching a large-mouth bass?

0.2, right? Right!

But, what is the probability of catching 3 bass out of 5 fish caught?"

Uh-oh....the neurons in this classroom have just started firing....and possibly, not down the correct paths!

The professor throws up this formula on the board:

I will use "typing notation" to restate this as

Pr(Y=k) = (nCk) * (p^k) * ((1-p)^(n-k)

Either way, the students' eyes go blurry and class is followed by a surge in optometrist appointments (possibly saving the day from a predicted sharp economic downturn)

But, the formula REALLY ISN'T COMPLICATED. It just looks that way. It's kinda' like those new-fangled cell phone PDA mp3 abc pdq xyz all-in-one devices. Hey, if you can find "Play" or "Send", it's not all that bad, right?

(Caution: I think the equation is blurry anyway but I can't tell...I'm not wearing my contacts. Please report any suspected true blurriness to the webmaster before visiting your optometrist.)

Moving forward...let's look at the first part:

Pr( Y=k) =

This is just saying that the probability that Y=3 is equal to something. What is Y=3? Well, you only have two options for binary variables....0 and 1. A 0 is usually meant to denote the "failure", the "miss", the "tails", the "no" while the 1 is usually meant to denote the "success", the "hit", the "heads", the "yes".

So, Pr(Y=3) is the probability of catching 3 bass....of having 3 successes....three hits....etc....is equal to something.

That something looks like a messy room full of n's , k's , and p's. (I thought I told you kids to clean...that...mess...up!)

Ok, let's do it.

First, we should befriend those letters. (If you don't believe that letters can be your friend then you definitely didn't watch enough Sesame Street as a kid!)

n is just our sample size....you should be ok with this by now. n=5

k is the number of Y=1's of interest. We want to know the probability of catching 3 bass out of 5. So, k=3

Finally, if we didn't know something about the probability of catching a bass then, yes, this problem would be VERY difficult to solve. But, we do know. (whew)

This probability is 0.2. So, p=.2

So, we plug in all these numbers and get the answer right? Right. But, that doesn't help you understand why the equation is so easy.

Let's look at the second part of the equation...(p^k).

This is asking you to raise the probability to the success power. So, we are raising .20 to the 3rd power. Why? Well, we already should know that the joint probability of two independent events is simply their product. For example, the probability of flipping two heads is .5 * .5 = .25.

So, what is the probability of buying 20 gallons of rotten milk when the probability is .05?

Well, that's .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05 * .05.

Now, why on Earth would we want to write all of that down when it's the same thing as saying .05 to the 20th power, which is .05^20? And, for that matter, why on Mars? Why on Jupiter? Why on any planet?

(I will save other possibilities such as "Why in Heaven?" and "Why in Hell" for the metaphysics community. It's just a bit too 'out there' for statisticians already dealing with issues of causality, etc...")

Many students solve p^k and turn that in as the answer. How callous, rude, and thoughtless! What about the other 2 fish? Y'know....those 2 non-bass?!? Do you really think that just because we are interested in the probability of 3 bass out of 5 fish that we can just ignore the 2 that aren't bass? If you ignore them, aren't you just talking about the probability of 3 bass out of 3 fish? (And discrimination of anything is SO pre-Y2K!)

There is a take-home point here! When we talk about the 0's, failures, misses, tails, non-bass, etc...we are not making a moral indictment on them. They are still INFORMATION. They still must be included in our calculations (even if we are upset that we only caught 3 bass and have 5 mouths at home to feed and don't have a cat to kick and feel compelled to blame Flipper).

So, this all ties into the third part of the equation...(1-p)^(n-k). Well, if the p=.2 is the probability of catching a bass then (1-.2), which is .8, is the probability of catching a non-bass. Like before, we wouldn't want to multiply .8 over and over and over by hand. If there are n=5 in the sample and there are k=3 bass, then we are now interested in 5-3 = 2 non-bass. So, we just raise .8 to the 2nd power.

Combining p^k and (1-p)^(n-k), all we are doing is asking for .2 * .2 * .2 * .8 * .8

It's just the joint probability solved by multiplying the individual probabilities for both successes AND failures. Why didn't they just Dr. Quantoid just put THAT in the textbook? Well, look back at the rotten milk example. Or think about this: What is the probability of 400 students out of 1,200 getting into graduate school when the probability is .14. Would you like to write that out? Or would it better to just say (.14^400) *((1-.14) ^ (1200 - 400))

Finally, what about the first part of the equation? Y'know that oversized parenthesis with an n on top of a k?
Well, we read that as "n choose k". So, it's 5 choose 3. Huh?

Of course we want to choose 3 from 5. But, guess what? There are a lot of ways to do this. Below are my fishing results for 3 different days.

Day 1: Bass Bass Bass Bluegill Carp
Day 2: Carp Bass Bass Shark Bass
Day 3: Whale Dolphin Bass Bass Bass

The point here is that there are many ways to catch 3 fish out of 5. It could be the first 3 fish as in day 1. It could be the last 3 fish as in day 3.

I could write out all of the possibilities and do some math but it's a lot easier to know that "n choose k" will give me the answer right away.

"n choose k" is equal to n! / k!(n-k!)

n! is (5*4*3*2*1)

k! is (3*2*1)

n-k! is (5-3)! = 2! = (2*1)

So, the full solution is ((5*4*3*2*1) / (3*2*1) * (2*1))

(Note: there was actually another bass inside of the whale but that violates the binomial assumption of identical trials so we'll pretend that didn't happen.)

(Note: After catching the whale, I almost caught a bass but the dolphin ate it....that violates the binomial assumption of independent trials so we'll pretend that didn't happen.)

So, what have we learned?

1. We are just multiplying the probabilities of "successes".
2. We are also multiplying the probabilities of "failures".
3. We are multiplying this by all the ways we can have those "successes".

Please send requests for more information to jeffmiller.research@gmail.com

Six (6) Common Causes of Autocorrelation

2007-10-24T17:28:00.000-04:00

This is a handy list of some of the causes of autocorrelation as distilled from Gujarati, D. N. (2002), Basic Econometrics

1. Omitted Variable Misspecification
2. Incorrect Functional Form Misspecification
3. Cobweb Phenomenon (it takes the public time to adjust to a change in policy)
4. Lags (yesterday impacts today)
5. Manipulation of Data (creating subsets over time resulting in a systematic pattern)
6. Data Transformations

Eight (8) Sources of Heteroscedasticity

2007-10-24T17:22:00.000-04:00

This is a handy list of some of the causes of heteroscedasticity (unequal conditional variances) as distilled from Gujarati, D. N. (2002), Basic Econometrics

1. Error Learning (variance declines as a result of practice)
2. Discretionary Capital (variance increases in the relationship between income and spending)
3. Improved Data Collection Procedures
4. Outliers
5. Misspecification (e.g., omitted variables)
6. Skewness
7. Incorrect Data Transformation
8. Incorrect Functional Form for the Model

Latent Growth Curve vs. Repeated Measures ANOVA

2007-10-01T18:26:00.000-04:00

I read this article last night that I think did an excellent job of explaining the advantages of latent growth curve (LGC) modeling over using repeated measures analysis of variance (RM-ANOVA). Below is my summary. My thoughts are consistent with theirs with the exception of Similarity 2 below (which they claim is peculiar to LGC).

SIMILARITY 1: Both result in the same parameter estimates (IFF the stricter RM-ANOVA assumptions are met).

SIMILARITY 2: Both permit a wide variety of alternative specifications such as modeling nonlinearity (ALTHOUGH this is much more straightforward in the LGCs).

DIFFERENCE 1: LGC estimates are not attenuated since the measurement error has been teased out. This is actually the primary advantage of all structural models for latent variables.

DIFFERENCE 2: LGC is not restricted to RM-ANOVAs compound symmetry or sphericity assumptions...any variance/covariance structured can be incorporated and tested.

DIFFERENCE 3: LGC won't listwise delete your missing data. It uses Full Information Maximum Likelihood (FIML) to incorporate all existing data.

DIFFERENCE 4: LGC isn't restricted to equally-spaced time intervals.

Article: Llabre, M. M., Spitzer, S., Siegel, S., Saab, P. G., & Schneiderman, N. (2004). Applying latent growth curve modeling to the investigation of individual differences in cardiovascular recovery from stress, Psychosomatic Medication, 66, 29-41.

Logistic Regression (in a nutshell)

2007-09-19T18:59:00.000-04:00

I recently googled "logistic regression" and was surprised at the variety of pages out there. I found ones with annotated output that didn't really explain the input. I found ones that explained logistic regression but bounced around from intuitive interpretations to throwing out statistical formulae. Even wikipedia provides just enough to let beginners think that they understand and then just enough to thoroughly confuse them.

Here is logistic regression in a nutshell:

1.) You have a dependent (response) variable that is binary. That means there only two outcomes. It's hit or miss. It's yes or no. It's on or off. It's male or female. And et cetera ad nauseam...

2.) We want to predict the probability of a response...so we are talking about a 0 to 1 scale.

3.) You shouldn't use linear regression for several reasons but the most common-sense reason is that linear regression can provide predictions NOT on a 0 to 1 scale. You could get a predicted probability of 2.4 or -8.3. That's just not possible (unless you are one of those mathematicians who likes to write proofs to debunk the status quo).

4.) We need a way to link the probabilistic response variable to the continuous and/or categorical predictors AND keep things on this 0 to 1 scale.

5.) Long story short....logistic regression winds up transforming the probabilities to odds and then taking the natural logarithm of these odds, which we now call logits. I'm not going to explain odds and logits. If you want to know more, then google "logistic regression", and you will get your fill.

6.) Suppose your response variable is passing a test (by convention, 0=no and 1=yes). You have 1 predictor - number of days present in class over the past 30 days. Suppose the regression coefficient (often just called beta) in the output is .14. You would then say that, on average, as class presence increases by 1 day, the natural logarithm of the odds of passing the test increases by .14.

7.) You thought I was going to make this easy understand, and now I am talking about natural logarithms of odds for my interpretation? No. For the interpretation, you can just talk about the odds. Most computer output will give you this number. Suppose the answer in odds is 1.24. Then, you just say that,on average, as class presence increases by 1 day, the odds of passing the test are multiplied by 1.24. In other words, for each additional day present, the odds of passing are 24% greater than that of not passing.

8.) To validate our findings, normally, we test whether the regression coefficient is equal to zero in the population. In logistic regression, the corresponding value for the odds is one (not zero). We got an odds of 1.24. Can we trust this? Or should we go with one (which would mean that the odds are the same for both passing and not passing, and hence class presence makes no difference at all)? Look at the p-value (significance). If it less than .05 (by convention), you have enough evidence to reject the notion that the odds are really one. You go ahead and support the 1.24 result.

We could talk at greater length about logistic regression concerns such as model fit, classification tables, its analog of discriminant analysis, the logistic regression versions of R2, etc...but that wouldn't be logistic regression in a nutshell.

The Validity of Validity

2007-09-17T18:53:00.000-04:00

I have received numerous requests on the lines of "Are my results valid?" and "Can I say that my survey is valid?". It is unfortunate that courses and texts continue to throw the validity word around as if it has some yes-no, right-wrong connotation. Hopefully, this blog will help dispel some of the myths.

1.) Validity is a matter of degree. There is no such thing as anything being perfectly valid or not valid. Even if this was proven false, it would only taken an application to a new population or the intervention of a historical event to debunk the claim. The best we can do is strive for the greatest validity possible.

2.) Related to this, whatever is being validated must be continuously re-validated. Some graduate students go to great length to show that their survey has been validated in umpteen contexts and scenarios. They reference factor analyses and instrument updates due to changes over time. This is great; however, the purpose of this is not only to validate the use of the instrument in their research but also to use their results to add to the body of evidence for validity of said instrument.

Imagine that you have a survey purported to measure "Fear of Terrorist Activity". First, dramatic events such as 9/11 would certainly require re-validation of that instrument. Second, constructs morph over time...the notion of terrorism has done so even between the 70's, 80's, 90's, and 00's. Third, there is always the possibility that the construct will differ when applied to a different population. For example, shouldn't we analyze the validity of this instrument when applying it to an indigenous tribe far removed from current terrorist events?

3.) We can talk about validity from either a micro or macro perspective. At the macro level, validity is based on the construct of interest. At the micro level, we can talk about specific "types" of validity including predictive, concurrent, criterion-related, face, content...and the list can go on much further than you would imagine.

4.) There is some disagreement about this but I have to agree with those that argue that validity has very little to do with a particular survey or test. We don't design a survey or test just for the fun of it. We design it because we are interested in measuring something such as self-esteem or intelligence. Once we design it, the survey or test itself does not stand up and announce results. People have to take the survey or test. At the end of the day, we have numbers. So, we should really talk about the validity of those numbers. If we were interested in measuring your heart rate, validity would be based on the the beats (the numbers).

But wait a second....very few surveys or tests are created just for the fun of getting a number. Ultimately, we DO SOMETHING with that number. Maybe we decide that Johnnie is smart enough for graduate school. Maybe Donna should be on medication for a disability. Maybe Richie should not be hired for this position. But when you use a survey (or rather the resulting numbers) to make these decisions, there is the risk that we made an incorrect decision (i.e., the Type I error). So, validity is really matter of degree to which we made an appropriate inference, conclusion, or decision from the number(s) that came from a test or survey designed to address a particular need. Does this mean we should ignore the content of the survey or the face appearance of the test? Of course, not. However, these don't truly answer questions about the degree of validity...they are crucial aspects of the design and procedures stages, and they have a direct impact on our conclusions regarding the degree of validity.

What can you draw from this? Be careful in how you use the word. If you want to sound like an amateur, then say, "Hey, I ran a confirmatory factor analysis. Guess what? My survey is valid". If you want sound more professional, then say, "Hey, I ran a confirmatory factor analysis. My results suggest that the original solution fits pretty well for my population. The original version has been used in many contexts and has had few negative consequences over the years. Based on my results, I should see the same thing happen. I've added evidence for greater validity in arguing my research conclusions, and I've contributed to the body of validity evidence for this survey."

Watch the video below to learn more about the impact of incorrect conclusions in statistical analyses.

I don't think we're in Kansas anymore, Toto!

2007-09-12T18:15:00.002-04:00

****************************************************************************
After meeting Glinda and the Munchkins, it's so incredibly clear. Just follow the yellow brick road right? If she just remembers those 5 magic words, she'll find her Emerald City and her free ride back to Kansas. Of course, it wasn't that easy. If it had been then the book would have been only a few chapters long. Instead, she decides to dance around with a scarecrow for awhile, slap a lion, take a nap in a poppy field, and go on a scenic tour of a Oz Witch Project.

Wow, this is starting to sound like the research process to me. We start with an idea. Then we meticulously determine our sample size and procedures. Finally, alas, all of that data is in a file itching to be analyzed. But at this point, my friend, we are most easily distracted from our yellow brick road.

Here's one scenario. Suppose you are researching the influence of an advertising campaign on the number of products purchased in one month for a brand new product. You decide it would be smart to look at the frequency distributions for the variables. (Yes, this is always a good thing to do.) But, uh-oh, your dependent variable is seriously positively skewed!

We talk to Fred who says, "Well, you really shouldn't assume a normal distribution...maybe you can normalize the distribution by taking the natural logarithm". Two weeks later, after debating the ethics of changing your numbers and accepting the notion of "linear transformations", you find out that you can't take the log anyway because you have a bunch of zeros.

So then you decide to go back to the company and say that you can't analyze this data because a bunch of people didn't buy the new product at all. And then you decide that doing that would probably not get you much future work for this business.

Mary suggests adding a very very small value such as .000000001 to all the numbers so that you can take the log of a now-transformed zero. But John then notes that you will now have an large number of ln(.000000001 ) values and that you should use a Hurdle or Zero-Inflated Poisson model.

Time to take a nap in the poppy field.

When you awaken, you realize that you only have 2 days left to meet the deadline. So, you decide to ignore that potential problem you wrestled with for 2 weeks. You move on to the predictor variables. You decide to run a factor analysis. Lo and behold, the factor structure is unexpected. Given the surprising results, you ask the client for a one-day extension. You run a cluster analysis on the sum of the scores making up the factors. You find two clusters. You decide to call them the "1=high on all factors" and "0=low on all factors". From there, you use this dichotomy in a latent class analysis. Finally, you take the 4 classes that emerge and regress them on........drum roll......the number of purchases made. (In case this has gotten confusing, the dependent variable has now become the predictor variable.)

Along comes the deadline. You send the client 137 or more pages of software printout and a bunch of graphs.

They look at this and say, "Wow...you did a lot of work. Now how does this answer our research question?"

You then realize that you aren't in Emerald City at all....this is looking more like the Oz Witch Project!

How can this be prevented? How can we sift through a myriad of analytic options and not lose track of the road before us?

#1. Stick a big note on your desk that says, "How does this help or distract me from answering the research question?"

#2. In the header and footer of all documents, type "The Effect of X on Y" or "The Relationship Between X and Y". Keep that research question clear and visible at all times.

#3. Write a quick sentence or two at the end of each analysis as if it were for the final report, and be sure to tie it into the original research questions.

#4. Tell your client, research partners, and/or graduate committee members how easy it is to lose sight of the forest for the trees. 80% of the solution to all problems is awareness.

#5. If you aren't a consultant, then find one. This is part of the consultant's job. A good consultant will not only know this but will help bring you back to the yellow brick road (or help you build a new one if necessary).

Please send requests for more information to jeffmiller.research@gmail.com