I was talking to another PhD student the other day that was presenting a schedule of the work that she was planning to do over the next few months. One project that she is deeply involved in at the moment in is analyzing data that has been collected from a set of real organizations. However, she also wants to then test these findings in the lab. I thought this might be a good opportunity to talk briefly about study design and what different kinds of research exist within social science.
Ideally, the research process goes in an order vaguely like this: A researcher comes up with an idea about how the world works, the relationship between some set of variables, etc. In most branches of social science, the researcher then creates a set of predictions about how different variables will be related to one another. This is less necessary in some fields such as (non-behavioral) economics. The next thing the researcher decides is what is the best way to determine if this relationship exists. Sometimes the question itself will inform what data should be used to test for an effect. If the question is, for example, about the relationship between stock price and employee stealing, then looking at a real organization may be ideal. Once a data source is identified, the researcher collects the data and does analysis on these data. After the researcher has interpreted the results, the work will go onto the publication process, either into a journal article, book, book chapter, or conference presentation.
I work in a very small world where I have used experiments in all of my work. My experiments, though not identical, have certain elements of design that I consistently use which adds familiarity to the design process for my studies. I know the manipulations and the kinds of acceptable tasks very well. Though the specifics have taken some time in the past to work out, I don't think it took me more than a few days to design each of the studies that I have used. The longest time has always been determining the task to use. The difficulty with tasks sometimes is the balance between creating a new, novel task that the participants won't be familiar with and choosing a task that has been tried and tested by you or your colleagues.
When I looked at the other student's schedule, I was genuinely surprised that she had 3 weeks scheduled for study design. When I talked to another student, she thought that 3 weeks was just about enough time. This interaction got me thinking why I was so surprised that the student chose such a long period of time to dedicate to study design. I don't think I am overly skilled at study design, but I could be using a different definition of study design than they were.
When a lab study is designed, the major decisions that have to be made are the task, the manipulations, and the measures. My manipulations have always been rather blunt and heavily tested: employee turnover or restricting communication. The manipulation of more delicate factors, such as feelings of group belonging, of fear, or feelings surrounding the exchange of favors, are, I imagine, much more difficult and may have smaller impacts on people. There are huge literatures investigating these factors, which may actually instead the time it takes to choose a manipulation because the researcher may feel like they need to be familiar with most of the prior work. I don't mean to come off as dismissive of other work, but if you spend all of your time reading all the published literature in your area, you'll never add to that literature yourself. It is a dangerous game of academia, unless you work along narrow specialties (which has been my strategy).
Once the core vision of a study has been determined and the three decisions mentioned earlier (task, manipulation, and measure) have been chosen, the materials have to be put together. I don't typically think of this as design, but it is a necessary part of the research process. This is the phase where study materials are drafted, the specifics of the task are decided, materials are purchased, and advertisement materials are readied. Another unsung, but important aspect of this process is the writing of a script. I was fortunate to have a reader on a student project strongly suggest I write one for my first solo project and graciously provided me an example. The script lists all the actions the experimenter does to prepare for the study, all the things the experimenter says, when things occur in relationship to one another, and the timeline of the study. Writing the script always has a way of highlighting to me glaring issues with the design of the study in both a shallow sense (operalization) and a deeper (theoretical) sense.
I hope this post provides you with some insight into the nuts and bolts of the social science research processes and may provide some tips to other scientists.
No comments:
Post a Comment