I Should be Shocked
That I should be shocked does not mean that I am.
Today, a 2009 graduate of the RSTC (Rhetoric, Scientific and Technical Writing) program posted a message via her "microblog" that she didn't know how to proceed with a survey she had conducted. "I'm not a statistician," her message concluded.
There are so many things wrong with this situation that I can't address them all, but I can certainly give an overview for those unfamiliar with the problem of humanities research attempting to be perceived as science. (I openly admit humanities are generally interpretive and observational, not mathematics. Nothing wrong with being observational and humanist.)
1. Surveys need to be composed with a knowledge of surveys.
If you do not understand survey methods, you should not administer a survey. The exception is a survey of purely demographic data, since it's hard to "bias" someone by asking his or her age, location, or eye color. But, if you are asking about opinions, personal experiences, recollections, et cetera, you should work with an expert in survey development. There are some proven techniques that must be used to avoid affecting the results.
A certain number of the questions should not relate to the intended topic. If participants know your purpose, they tailor their answers accordingly. It's bad methodology to state the purpose of your survey if you are studying opinions or memories.
Questions should be asked in two or more forms. The repetition, with changed wording, helps smooth out possible reactions to the words in a question. A common trick is "reversing" some questions; you invert the scale or responses on these questions before analyzing the data.
There are far more points to make, but ideally you get the point: serious surveys are by necessity longer and more complex.
2. Data should be analyzed with a knowledge of statistics
I'd really hope this is self-evident, but it apparently is not to some in the humanities. While you can generate mean, mode, and median data, you also need to know which of these matter and why. You need to know how and why correlations are calculated using various models. Unfortunately, modern polling data has left many people with the impression that the point of a survey is to find "X percent of people in group Y believe A, B, and C." It's not that simple.
If we want our graduate degrees to be taken seriously, the students should probably take a year of statistics -- maybe more. No one should be using quantitative data without understanding the analyses of such data.
Today, a 2009 graduate of the RSTC (Rhetoric, Scientific and Technical Writing) program posted a message via her "microblog" that she didn't know how to proceed with a survey she had conducted. "I'm not a statistician," her message concluded.
There are so many things wrong with this situation that I can't address them all, but I can certainly give an overview for those unfamiliar with the problem of humanities research attempting to be perceived as science. (I openly admit humanities are generally interpretive and observational, not mathematics. Nothing wrong with being observational and humanist.)
1. Surveys need to be composed with a knowledge of surveys.
If you do not understand survey methods, you should not administer a survey. The exception is a survey of purely demographic data, since it's hard to "bias" someone by asking his or her age, location, or eye color. But, if you are asking about opinions, personal experiences, recollections, et cetera, you should work with an expert in survey development. There are some proven techniques that must be used to avoid affecting the results.
A certain number of the questions should not relate to the intended topic. If participants know your purpose, they tailor their answers accordingly. It's bad methodology to state the purpose of your survey if you are studying opinions or memories.
Questions should be asked in two or more forms. The repetition, with changed wording, helps smooth out possible reactions to the words in a question. A common trick is "reversing" some questions; you invert the scale or responses on these questions before analyzing the data.
There are far more points to make, but ideally you get the point: serious surveys are by necessity longer and more complex.
2. Data should be analyzed with a knowledge of statistics
I'd really hope this is self-evident, but it apparently is not to some in the humanities. While you can generate mean, mode, and median data, you also need to know which of these matter and why. You need to know how and why correlations are calculated using various models. Unfortunately, modern polling data has left many people with the impression that the point of a survey is to find "X percent of people in group Y believe A, B, and C." It's not that simple.
If we want our graduate degrees to be taken seriously, the students should probably take a year of statistics -- maybe more. No one should be using quantitative data without understanding the analyses of such data.
Comments
Post a Comment