Analyzing data is quite an odd experience in the research world. In statistics classes, you learn about a lot of complicated models, tests, and assumptions. But, my experience analyzing data from experiments is that much of what is learned in the classroom is ignored. That isn't to say that I willingly defy the instruction I received in my classes. Instead it is that the things I learned in the majority of my stats classes are not that important for studying experiments.
Why is there this disconnect? First, most experiments are technically immune from a lot of the potential problems that stats classes teach you about. If there is random assignment to condition then individual differences shouldn't matter. Manipulations and some dependent measures can be thought of as perfect measures because they are the thing itself. I don't have a huge amount of experience in this area but the data I have typically gotten from experiments does not typically violate assumptions or is unable to violate the assumptions on ANOVA or regression such as independence. A social psychologist once implied to me that, in our field, if you use fancy statistical techniques or describe all of the tests for assumptions that you ran on the data that it can make any effects less believable. The researcher proposed this because most of the effects we investigate are measurable by ANOVA or linear regression. The researcher may be using fancy statistics because that is the only occasion when the effect exists.
This is an interesting situation because, if this is really the case, it suggests that at least part of social psychology is unwilling to accept advances in statistical procedures or statistical rigor because they don't want to be seen as hiding behind the math. If a paper doesn't use one of a small handful of methods then they are open to criticism for their statistical methods. If they use simple analyses, however, they are less open to complaints about their statistics. This may not be the true case or the case in the majority of social psychology, but I have reason to believe that it exists. It is also certainly true that a sign of experimenter 'p-hacking' is the use of convoluted analyses that may not be entirely appropriate, leading to spurious effects.
I don't mean to suggest that new innovations never make it into social psychological research. Preacher and Hayes have made a huge splash in the psychological community by introducing a way to more accurately gauge the existence and the effect size for many kinds of statistical mediations. I think that a partial reason for this acceptance, however, was their demonstration that the traditional ways of testing for mediations were more likely than their method to say that there is no mediation when a mediation does actually exist. This fact made the acceptance of a new method more appealing to the community at large, partially because it is more accurate and especially because it is more accurate in such a direction that mediations that were previously not supported may now be supported.
It is an interesting world that I honestly do not know much about. As far as journal publications go, if the editor and reviewers (normally no more than 5 people in total) think the stats you use are okay then your work can be published. If the stats are easier to understand, then your work is more likely to be published. Also, if your stats are very hard to understand because the methods are very obscure or new can also lead to your work being published. Though there were multiple issues with Daryl Bem's 2011 paper in JPSP (a very prestigious journal), but one criticism was that the stats he used were too complex and picked up on subtle, random differences. I think that the analytical world I live in is very interesting, but I just don't understand it sometimes.
Wednesday, April 16, 2014
Tuesday, April 15, 2014
Turnover and enactment of change
Describe Kane et al (2005) and maybe Levine & Choi (2009)
In much of the literature about turnover, it is unclear how newcomers are influencing the outcomes for the groups that they join. More recent literature has attempted to categorize the ways that newcomers adapt to the groups they join and how groups adapt to the newcomers. In the study I describe today, the researchers were curious what factors influence whether a newcomer is able or willing to share their ideas with the rest of the group. There is an assumption in much of the management literature that newcomer's primary value is in the new ideas they bring to the group. But, under what conditions that occurs is less clear. It certainly is not as often as possible or there would be much more value in the world.
The study I want to describe is Kane, Argote, & Levine (2005). These researchers decided to use an experiment to investigate some of their ideas using the frame of social identification. The researchers proposed that if group member shared a common social identity with a newcomer that they would be more willing to accept new ideas into the group. Unfortunately, this study was not able to determine directionality (whether the effect is newcomers' willingness to give ideas or oldtimers' willingness to accept them) but this step was a great step forward in this research.
In this task, the participants made paper boats in assembly lines. The researchers demonstrated how paper boats could be made but were clear that their requirement was to make as many paper boats that fit the requirements for the task and not just this specific boat. Some groups learned a method of making paper boats that required 7 folds while other groups learned a method that took 12 folds. Though the group with the smaller number of folds had one fold that was somewhat complex, they were much more efficient in general than groups with 12 folds (based on pretesting). The groups were told to use an assembly line to construct the boats and the more difficult fold is done by the middle member.
The other manipulation in the study was whether the groups shared a sense of collective social identity with each other. In each experimental session, 2 groups were brought into the lab at the same time. They both also participated in a training period in the same room. However, in the high social identity condition, the groups were given the same names, seated in an integrated fashion differently, and given a reward scheme where the performance of both groups would lead to better outcomes for all the individuals. In the low social identity condition, these three factors were changed so the groups seemed less similar to one another and their reward was not interdependent with the other team.
The other action in the study that the experimenters made was very clever. The middle members for each group switched from participating with one group to participating in another group. Therefore, for some groups the new member had the same experience as the group they are entering (experience with low or high efficiency folding techniques) whereas for other groups there was a difference (the new member has the low or high efficiency folding technique but the group they join has the opposite). The new member is therefore in a position where they need to learn the technique the group is using, or the member needs to try and get the group to accept the way of doing the task that they are most used to.
Skipping ahead to the results, almost no groups accepted the newcomers folding strategy if the strategy was worse than the one that they already have. There was also a main effect of the identity. Shen the groups had a shared identity, they were much more likely to accept the new member's strategy into their group. If the groups shared an identity (from having the same group name and having a shared reward structure, then they accepted the new member's better way of constructing the boat about 70% of the time. If they didn't share that same identity, however, then the group didn't accept the new member's superior way of making the boat that often (only 25% of the time).
The results for performance were a bit harder to interpret. All groups performed better over time, generating more boats in the last trial than in the first one. But there wasn't a strong direct relationship of the new member having a superior routine and performance. The researchers found that when the new member introduced a better routine to the group, that they experienced a larger increase in their performance than when the new member has a worse routine. These differences, however, are only for groups that share an identity with the new member. If the group didn't share an identity with the new member, then it didn't matter whether the new member had a better or worse routine, partially because these groups accepted that routine infrequently.
In context, this study was very significant for a few reasons. First, Argote had done significant work on learning within groups and organizations. This, however, was one of the first study that demonstrates both how learning can occur within a group and that there are certain variables that influence whether a group can learn from a new member. The variable of interet here was social identity but other work has looked at a lot of other factors (see Rink et al., 2013 or a review). Second, this study demonstrated that groups have the ability to recognize advantageous strategies and use them. This was demonstrated in some earlier work by McGrath, but Kane's study is a very clean experimental setting. Lastly, the results suggest that learning new strategies can be costly to a group, hence the small differences in performance for groups where the new member had a better routine compared to groups that received a new member with a less efficient routine.
In much of the literature about turnover, it is unclear how newcomers are influencing the outcomes for the groups that they join. More recent literature has attempted to categorize the ways that newcomers adapt to the groups they join and how groups adapt to the newcomers. In the study I describe today, the researchers were curious what factors influence whether a newcomer is able or willing to share their ideas with the rest of the group. There is an assumption in much of the management literature that newcomer's primary value is in the new ideas they bring to the group. But, under what conditions that occurs is less clear. It certainly is not as often as possible or there would be much more value in the world.
The study I want to describe is Kane, Argote, & Levine (2005). These researchers decided to use an experiment to investigate some of their ideas using the frame of social identification. The researchers proposed that if group member shared a common social identity with a newcomer that they would be more willing to accept new ideas into the group. Unfortunately, this study was not able to determine directionality (whether the effect is newcomers' willingness to give ideas or oldtimers' willingness to accept them) but this step was a great step forward in this research.
In this task, the participants made paper boats in assembly lines. The researchers demonstrated how paper boats could be made but were clear that their requirement was to make as many paper boats that fit the requirements for the task and not just this specific boat. Some groups learned a method of making paper boats that required 7 folds while other groups learned a method that took 12 folds. Though the group with the smaller number of folds had one fold that was somewhat complex, they were much more efficient in general than groups with 12 folds (based on pretesting). The groups were told to use an assembly line to construct the boats and the more difficult fold is done by the middle member.
The other manipulation in the study was whether the groups shared a sense of collective social identity with each other. In each experimental session, 2 groups were brought into the lab at the same time. They both also participated in a training period in the same room. However, in the high social identity condition, the groups were given the same names, seated in an integrated fashion differently, and given a reward scheme where the performance of both groups would lead to better outcomes for all the individuals. In the low social identity condition, these three factors were changed so the groups seemed less similar to one another and their reward was not interdependent with the other team.
The other action in the study that the experimenters made was very clever. The middle members for each group switched from participating with one group to participating in another group. Therefore, for some groups the new member had the same experience as the group they are entering (experience with low or high efficiency folding techniques) whereas for other groups there was a difference (the new member has the low or high efficiency folding technique but the group they join has the opposite). The new member is therefore in a position where they need to learn the technique the group is using, or the member needs to try and get the group to accept the way of doing the task that they are most used to.
Skipping ahead to the results, almost no groups accepted the newcomers folding strategy if the strategy was worse than the one that they already have. There was also a main effect of the identity. Shen the groups had a shared identity, they were much more likely to accept the new member's strategy into their group. If the groups shared an identity (from having the same group name and having a shared reward structure, then they accepted the new member's better way of constructing the boat about 70% of the time. If they didn't share that same identity, however, then the group didn't accept the new member's superior way of making the boat that often (only 25% of the time).
The results for performance were a bit harder to interpret. All groups performed better over time, generating more boats in the last trial than in the first one. But there wasn't a strong direct relationship of the new member having a superior routine and performance. The researchers found that when the new member introduced a better routine to the group, that they experienced a larger increase in their performance than when the new member has a worse routine. These differences, however, are only for groups that share an identity with the new member. If the group didn't share an identity with the new member, then it didn't matter whether the new member had a better or worse routine, partially because these groups accepted that routine infrequently.
In context, this study was very significant for a few reasons. First, Argote had done significant work on learning within groups and organizations. This, however, was one of the first study that demonstrates both how learning can occur within a group and that there are certain variables that influence whether a group can learn from a new member. The variable of interet here was social identity but other work has looked at a lot of other factors (see Rink et al., 2013 or a review). Second, this study demonstrated that groups have the ability to recognize advantageous strategies and use them. This was demonstrated in some earlier work by McGrath, but Kane's study is a very clean experimental setting. Lastly, the results suggest that learning new strategies can be costly to a group, hence the small differences in performance for groups where the new member had a better routine compared to groups that received a new member with a less efficient routine.
Monday, April 14, 2014
The research process and study design
I was talking to another PhD student the other day that was presenting a schedule of the work that she was planning to do over the next few months. One project that she is deeply involved in at the moment in is analyzing data that has been collected from a set of real organizations. However, she also wants to then test these findings in the lab. I thought this might be a good opportunity to talk briefly about study design and what different kinds of research exist within social science.
Ideally, the research process goes in an order vaguely like this: A researcher comes up with an idea about how the world works, the relationship between some set of variables, etc. In most branches of social science, the researcher then creates a set of predictions about how different variables will be related to one another. This is less necessary in some fields such as (non-behavioral) economics. The next thing the researcher decides is what is the best way to determine if this relationship exists. Sometimes the question itself will inform what data should be used to test for an effect. If the question is, for example, about the relationship between stock price and employee stealing, then looking at a real organization may be ideal. Once a data source is identified, the researcher collects the data and does analysis on these data. After the researcher has interpreted the results, the work will go onto the publication process, either into a journal article, book, book chapter, or conference presentation.
I work in a very small world where I have used experiments in all of my work. My experiments, though not identical, have certain elements of design that I consistently use which adds familiarity to the design process for my studies. I know the manipulations and the kinds of acceptable tasks very well. Though the specifics have taken some time in the past to work out, I don't think it took me more than a few days to design each of the studies that I have used. The longest time has always been determining the task to use. The difficulty with tasks sometimes is the balance between creating a new, novel task that the participants won't be familiar with and choosing a task that has been tried and tested by you or your colleagues.
When I looked at the other student's schedule, I was genuinely surprised that she had 3 weeks scheduled for study design. When I talked to another student, she thought that 3 weeks was just about enough time. This interaction got me thinking why I was so surprised that the student chose such a long period of time to dedicate to study design. I don't think I am overly skilled at study design, but I could be using a different definition of study design than they were.
When a lab study is designed, the major decisions that have to be made are the task, the manipulations, and the measures. My manipulations have always been rather blunt and heavily tested: employee turnover or restricting communication. The manipulation of more delicate factors, such as feelings of group belonging, of fear, or feelings surrounding the exchange of favors, are, I imagine, much more difficult and may have smaller impacts on people. There are huge literatures investigating these factors, which may actually instead the time it takes to choose a manipulation because the researcher may feel like they need to be familiar with most of the prior work. I don't mean to come off as dismissive of other work, but if you spend all of your time reading all the published literature in your area, you'll never add to that literature yourself. It is a dangerous game of academia, unless you work along narrow specialties (which has been my strategy).
Once the core vision of a study has been determined and the three decisions mentioned earlier (task, manipulation, and measure) have been chosen, the materials have to be put together. I don't typically think of this as design, but it is a necessary part of the research process. This is the phase where study materials are drafted, the specifics of the task are decided, materials are purchased, and advertisement materials are readied. Another unsung, but important aspect of this process is the writing of a script. I was fortunate to have a reader on a student project strongly suggest I write one for my first solo project and graciously provided me an example. The script lists all the actions the experimenter does to prepare for the study, all the things the experimenter says, when things occur in relationship to one another, and the timeline of the study. Writing the script always has a way of highlighting to me glaring issues with the design of the study in both a shallow sense (operalization) and a deeper (theoretical) sense.
I hope this post provides you with some insight into the nuts and bolts of the social science research processes and may provide some tips to other scientists.
Ideally, the research process goes in an order vaguely like this: A researcher comes up with an idea about how the world works, the relationship between some set of variables, etc. In most branches of social science, the researcher then creates a set of predictions about how different variables will be related to one another. This is less necessary in some fields such as (non-behavioral) economics. The next thing the researcher decides is what is the best way to determine if this relationship exists. Sometimes the question itself will inform what data should be used to test for an effect. If the question is, for example, about the relationship between stock price and employee stealing, then looking at a real organization may be ideal. Once a data source is identified, the researcher collects the data and does analysis on these data. After the researcher has interpreted the results, the work will go onto the publication process, either into a journal article, book, book chapter, or conference presentation.
I work in a very small world where I have used experiments in all of my work. My experiments, though not identical, have certain elements of design that I consistently use which adds familiarity to the design process for my studies. I know the manipulations and the kinds of acceptable tasks very well. Though the specifics have taken some time in the past to work out, I don't think it took me more than a few days to design each of the studies that I have used. The longest time has always been determining the task to use. The difficulty with tasks sometimes is the balance between creating a new, novel task that the participants won't be familiar with and choosing a task that has been tried and tested by you or your colleagues.
When I looked at the other student's schedule, I was genuinely surprised that she had 3 weeks scheduled for study design. When I talked to another student, she thought that 3 weeks was just about enough time. This interaction got me thinking why I was so surprised that the student chose such a long period of time to dedicate to study design. I don't think I am overly skilled at study design, but I could be using a different definition of study design than they were.
When a lab study is designed, the major decisions that have to be made are the task, the manipulations, and the measures. My manipulations have always been rather blunt and heavily tested: employee turnover or restricting communication. The manipulation of more delicate factors, such as feelings of group belonging, of fear, or feelings surrounding the exchange of favors, are, I imagine, much more difficult and may have smaller impacts on people. There are huge literatures investigating these factors, which may actually instead the time it takes to choose a manipulation because the researcher may feel like they need to be familiar with most of the prior work. I don't mean to come off as dismissive of other work, but if you spend all of your time reading all the published literature in your area, you'll never add to that literature yourself. It is a dangerous game of academia, unless you work along narrow specialties (which has been my strategy).
Once the core vision of a study has been determined and the three decisions mentioned earlier (task, manipulation, and measure) have been chosen, the materials have to be put together. I don't typically think of this as design, but it is a necessary part of the research process. This is the phase where study materials are drafted, the specifics of the task are decided, materials are purchased, and advertisement materials are readied. Another unsung, but important aspect of this process is the writing of a script. I was fortunate to have a reader on a student project strongly suggest I write one for my first solo project and graciously provided me an example. The script lists all the actions the experimenter does to prepare for the study, all the things the experimenter says, when things occur in relationship to one another, and the timeline of the study. Writing the script always has a way of highlighting to me glaring issues with the design of the study in both a shallow sense (operalization) and a deeper (theoretical) sense.
I hope this post provides you with some insight into the nuts and bolts of the social science research processes and may provide some tips to other scientists.
Wednesday, April 9, 2014
Goodman
Coal miner study
One part of my graduate education that I count as one of the most fortunate was the limited amount of experience I had interacting with Professor Paul Goodman. Paul Goodman unfortunately passed away shortly after I passed my qualifying exams. Paul Goodman was an extremely interesting and committed researcher that allowed his personal feelings of justice influence the direction of his work in a very real way without allowing them to cloud the scientific process. Paul was truely one of a kind.
After Paul passed away, I spent some time talking to his wife and children as they discussed his upbringing and what motivated some of this work. From what I recall, both of his parents were liberal social activists in New England. From an early age they instilled in Paul that organizations have a responsibility to treat their employees well. Though I'm sure many other things influenced his choice of career, Paul eventually began studying the ways that employees interact with management in organizations. Paul was an avid film-maker who did a series of videos about the current state and future of work. He typically interviewed average people in industries that were changing. Many of these films can be found at a permanent collection at Carnegie Mellon's library website: http://dli.library.cmu.edu/paulgoodman/
The last two projects that I know of Paul perusing was a long-term project on the science of science teams. Though I do not know his specific motivation, scientists often apply much less social science to their organization than what we actually know. After Paul died, this project dissolved due to the cohesive power of Paul's personality disappearing. The other was a more amorphous process that I think perfectly sums up Paul's outlook on the world. He and his assistants conducted hundreds of long-form interviews asking average people what they thought the American dream was, if they strove for it, and what kind of world they wanted for their children.
Though Paul completed a lot of interesting work, what I'd like to talk about today was some of the work that came out of his multi-year coal-mining project. In this work, Paul went to coal mines in the mid-Atlantic and interviewed miners in their place of work. By that I mean underground in the mine itself. Paul told me on multiple occasions that he thought that the ability to conduct the interviews and collect data in the mine itself gave him a much more accurate perception of what it was like to work in this environment. My father, who is from Pennsylvania described to me when I was very young that my great grandfather's worked in a coal mine. This profession and the work that Paul did therefore always seemed to touch me a bit closer as I always imagined my great grandfather in the place of the miners in the papers.
The paper I would like to describe of Paul's is one that he wrote with Dennis Leyden. This work was supported by the U.S. Bureau of Mines. I can only make a guess but I think that the Bureau were interested in how the relationships between the individual workers in the mine were related to mine outcomes. Mines vary in productivity enormously and one possible reason is the kinds of relationships the individual workers have with one another. Goodman and Leyden proposed that the mines provided a good opportunity to look at the effects of familiarity on the small teams that work together within a coal mine. (In a prior study, the researchers had already identified that an individual with little familiarity with a mine was more likely to have an accident.)
Mining crews were sets of workers doing one of three unique roles. Those roles were: the miner operator, the bolter, and the car operator. Each crew typically had a pair of people performing each role. Though each role is unique and there are skills associated with the roles, the authors argue that the specific strategies the individuals use vary from crew to crew based on personal differences and the features of the part of the mine the group is in. Though the researchers do not mention it specifically in this paper, another factor that I imagine is important is the cognitive interdependence of the individuals on one another.
Without getting into too much analytical detail, the researchers used information about which groups individual were working on to create a measure of whether individuals had worked with one another before and to what extent a given group's members were familiar with one another. Overall, the researchers found that the levels of familiarity between the group members was predictive of the overall mine productivity. They found some evidence that different kinds of familiarity mattered more than others but they felt that overall familiarity mattered more.
In rereading this paper, I found myself reminded of some other interesting work by Karl Weick on aircrews. Like I am attempting to do in this blog, Weick preferred description over analytics and wrote extremely provoking papers based on his reading and observations of real events. Weick's observation of air crews found very similar effects of familiarity on the air crews ability to perform without errors. I'm sure I will discuss some of his other work later in this blog.
One part of my graduate education that I count as one of the most fortunate was the limited amount of experience I had interacting with Professor Paul Goodman. Paul Goodman unfortunately passed away shortly after I passed my qualifying exams. Paul Goodman was an extremely interesting and committed researcher that allowed his personal feelings of justice influence the direction of his work in a very real way without allowing them to cloud the scientific process. Paul was truely one of a kind.
After Paul passed away, I spent some time talking to his wife and children as they discussed his upbringing and what motivated some of this work. From what I recall, both of his parents were liberal social activists in New England. From an early age they instilled in Paul that organizations have a responsibility to treat their employees well. Though I'm sure many other things influenced his choice of career, Paul eventually began studying the ways that employees interact with management in organizations. Paul was an avid film-maker who did a series of videos about the current state and future of work. He typically interviewed average people in industries that were changing. Many of these films can be found at a permanent collection at Carnegie Mellon's library website: http://dli.library.cmu.edu/paulgoodman/
The last two projects that I know of Paul perusing was a long-term project on the science of science teams. Though I do not know his specific motivation, scientists often apply much less social science to their organization than what we actually know. After Paul died, this project dissolved due to the cohesive power of Paul's personality disappearing. The other was a more amorphous process that I think perfectly sums up Paul's outlook on the world. He and his assistants conducted hundreds of long-form interviews asking average people what they thought the American dream was, if they strove for it, and what kind of world they wanted for their children.
Though Paul completed a lot of interesting work, what I'd like to talk about today was some of the work that came out of his multi-year coal-mining project. In this work, Paul went to coal mines in the mid-Atlantic and interviewed miners in their place of work. By that I mean underground in the mine itself. Paul told me on multiple occasions that he thought that the ability to conduct the interviews and collect data in the mine itself gave him a much more accurate perception of what it was like to work in this environment. My father, who is from Pennsylvania described to me when I was very young that my great grandfather's worked in a coal mine. This profession and the work that Paul did therefore always seemed to touch me a bit closer as I always imagined my great grandfather in the place of the miners in the papers.
The paper I would like to describe of Paul's is one that he wrote with Dennis Leyden. This work was supported by the U.S. Bureau of Mines. I can only make a guess but I think that the Bureau were interested in how the relationships between the individual workers in the mine were related to mine outcomes. Mines vary in productivity enormously and one possible reason is the kinds of relationships the individual workers have with one another. Goodman and Leyden proposed that the mines provided a good opportunity to look at the effects of familiarity on the small teams that work together within a coal mine. (In a prior study, the researchers had already identified that an individual with little familiarity with a mine was more likely to have an accident.)
Mining crews were sets of workers doing one of three unique roles. Those roles were: the miner operator, the bolter, and the car operator. Each crew typically had a pair of people performing each role. Though each role is unique and there are skills associated with the roles, the authors argue that the specific strategies the individuals use vary from crew to crew based on personal differences and the features of the part of the mine the group is in. Though the researchers do not mention it specifically in this paper, another factor that I imagine is important is the cognitive interdependence of the individuals on one another.
Without getting into too much analytical detail, the researchers used information about which groups individual were working on to create a measure of whether individuals had worked with one another before and to what extent a given group's members were familiar with one another. Overall, the researchers found that the levels of familiarity between the group members was predictive of the overall mine productivity. They found some evidence that different kinds of familiarity mattered more than others but they felt that overall familiarity mattered more.
In rereading this paper, I found myself reminded of some other interesting work by Karl Weick on aircrews. Like I am attempting to do in this blog, Weick preferred description over analytics and wrote extremely provoking papers based on his reading and observations of real events. Weick's observation of air crews found very similar effects of familiarity on the air crews ability to perform without errors. I'm sure I will discuss some of his other work later in this blog.
Tuesday, April 8, 2014
Coginitive Interdependence [Deep Dive] - Moreland series - Part 3
This post describes the studies commissioned by the Army that Levine et al. explored. In Part 2, the studies on productivity were explored. In this post, I focus on the experiments about innovation.
Creativity Experiment 1 -Assigned and/or maligned (published as Choi & Levine, 2004)
The researchers then shifted away from performance as the primary variable of interest and into the effect of turnover on group innovation. These studies used a air-surveillance task that John Levine and his students have used in several papers that I know about. Groups work together to monitor the radar at a base and assign threat levels to the different radar contacts. In each of the three member groups, 2 individuals were specialists and 1 acted as the commander. The specialists essentially collect information about the radar contacts and the commander receives that information and is tasked with making a decision. There were two different strategies that could be used in the strategies to collect information that varied on whether the importance of the information the specialists collected was the same for both or whether the difficulty of getting the information was the same for both.
In the first experiment, the researchers manipulated whether the group was able to choose their strategy and how well their feedback suggested that they had performed. In the experimental setup, the group was either assigned one of the two strategies above or they were allowed to choose one. Then the group performed the task. Half the groups were told that they had performed well and the other half were told that they had performed below a passing rate. One of the specialists was then chosen and replaced with a confederate. In social psych research, a confederate is someone who pretends to be a normal participant but has been coached to act in a particular way. The newcomer then proposed that the group switch to the opposite strategy of whichever they had chosen in the first trial.
The researchers used whether the group accepted or rejected the strategy the newcomer proposed as the variable of interest. Because this could be affected by a multitude of factors, the researchers measured how committed the members were to the previous strategy, how much they liked the team, performance in the first trial, etc. The researchers found results that were inline with what they anticipated. If groups were told they failed to perform well in the first trial, they were more willing to accept the newcomers idea. The groups were also more likely to accept the idea of the newcomer if the group had not been allowed to choose their own strategy.
The researchers then did some additional analyses and proposed what led to the group's receptivity to the newcomer's proposal. The two variables the researchers proposed mediate the effect of team choice on the acceptance of the newcomer: commitment and perceived performance. If the group had a choice in their strategy, they were more committed to their strategy and they perceived their performance as better. The researchers were fairly satisfied in these findings but they also thought that the way the newcomer proposed their innovative idea likely would have an effect on whether the group accepted it. This led to the second experiment.
Creativity Experiment 2 -An Assertive Story
This study was run very similarly to the first creativity study except that the kind of language the newcomer used was varied. As before, groups are more likely to accept the newcomer's innovation when the group was told that they had failed in the first trial. There was also what is called a statistically significant interaction. An interaction just means that whether one variable has an influence depends on another variable. When the groups were told that they had succeeded, it did not matter whether the newcomer was assertive or not, the acceptance rate was always about 45%. If the group had been told they failed, however, they were more likely to accept the ideas of the newcomer if the newcomer was assertive (~85%) versus if the newcomer was not assertive (~60%). [Note: this effect is only 'marginally significant' meaning that our confidence in the effect is not overly high.] The researchers had hoped for stronger effects but still thought this study was valuable.
Computational simulations, the shallowest dive
The last part of the technical report provides some information about a series of computational simulations that were included in this project. Very briefly, a computational simulation puts a bunch of agents into a box. Each agent represents a person, organization, etc. The agents are given some rules to live by, some of which may vary systematically (share information with another agent if they are within 2 spaces vs. share information with another agent if they occupy the same space). There is also a level of randomness that is added to the agents decisions to help simulate the real world. Simulations are becoming more and more accepted within management-type research though I am not sure how accepted they are within general social psychology.
In the series of simulations presented in the report, the authors focus on the effect of transactive memory and changes in the environment. In the first simulation, the researchers find some evidence that suggsts that the value of a transactive memory is curvilinear with the size of the group. They found that if the group is fairly small, the difference in speed to completion of a task by the agents was about the same regardless of whether the group had a transactive memory or not. There was a definite benefit of TMS when groups were larger (between 15 and 27), but the benefit reduced for larger groups (35). I personally think that this is an artifact of how the agent's task is structured, but it does seem fairly reasonable. The last simulation suggested that transactive memory is particularly useful if the group completes multiple different kinds of tasks that are completed in alternating order. A transactive memory allows the groups to more quickly shift tasks, leading to a consistency in time to completion.
Though the studies in this report were not all successful, I found it particularly interesting. The ability to try out new ideas that this study provided also certainly helped the researchers develop their later studies and directed other researchers toward these topics.
I think this post completes my sequence on cognitive interdependence for now, though I'm sure it will crop back up :P
Creativity Experiment 1 -Assigned and/or maligned (published as Choi & Levine, 2004)
The researchers then shifted away from performance as the primary variable of interest and into the effect of turnover on group innovation. These studies used a air-surveillance task that John Levine and his students have used in several papers that I know about. Groups work together to monitor the radar at a base and assign threat levels to the different radar contacts. In each of the three member groups, 2 individuals were specialists and 1 acted as the commander. The specialists essentially collect information about the radar contacts and the commander receives that information and is tasked with making a decision. There were two different strategies that could be used in the strategies to collect information that varied on whether the importance of the information the specialists collected was the same for both or whether the difficulty of getting the information was the same for both.
In the first experiment, the researchers manipulated whether the group was able to choose their strategy and how well their feedback suggested that they had performed. In the experimental setup, the group was either assigned one of the two strategies above or they were allowed to choose one. Then the group performed the task. Half the groups were told that they had performed well and the other half were told that they had performed below a passing rate. One of the specialists was then chosen and replaced with a confederate. In social psych research, a confederate is someone who pretends to be a normal participant but has been coached to act in a particular way. The newcomer then proposed that the group switch to the opposite strategy of whichever they had chosen in the first trial.
The researchers used whether the group accepted or rejected the strategy the newcomer proposed as the variable of interest. Because this could be affected by a multitude of factors, the researchers measured how committed the members were to the previous strategy, how much they liked the team, performance in the first trial, etc. The researchers found results that were inline with what they anticipated. If groups were told they failed to perform well in the first trial, they were more willing to accept the newcomers idea. The groups were also more likely to accept the idea of the newcomer if the group had not been allowed to choose their own strategy.
The researchers then did some additional analyses and proposed what led to the group's receptivity to the newcomer's proposal. The two variables the researchers proposed mediate the effect of team choice on the acceptance of the newcomer: commitment and perceived performance. If the group had a choice in their strategy, they were more committed to their strategy and they perceived their performance as better. The researchers were fairly satisfied in these findings but they also thought that the way the newcomer proposed their innovative idea likely would have an effect on whether the group accepted it. This led to the second experiment.
Creativity Experiment 2 -An Assertive Story
This study was run very similarly to the first creativity study except that the kind of language the newcomer used was varied. As before, groups are more likely to accept the newcomer's innovation when the group was told that they had failed in the first trial. There was also what is called a statistically significant interaction. An interaction just means that whether one variable has an influence depends on another variable. When the groups were told that they had succeeded, it did not matter whether the newcomer was assertive or not, the acceptance rate was always about 45%. If the group had been told they failed, however, they were more likely to accept the ideas of the newcomer if the newcomer was assertive (~85%) versus if the newcomer was not assertive (~60%). [Note: this effect is only 'marginally significant' meaning that our confidence in the effect is not overly high.] The researchers had hoped for stronger effects but still thought this study was valuable.
Computational simulations, the shallowest dive
The last part of the technical report provides some information about a series of computational simulations that were included in this project. Very briefly, a computational simulation puts a bunch of agents into a box. Each agent represents a person, organization, etc. The agents are given some rules to live by, some of which may vary systematically (share information with another agent if they are within 2 spaces vs. share information with another agent if they occupy the same space). There is also a level of randomness that is added to the agents decisions to help simulate the real world. Simulations are becoming more and more accepted within management-type research though I am not sure how accepted they are within general social psychology.
In the series of simulations presented in the report, the authors focus on the effect of transactive memory and changes in the environment. In the first simulation, the researchers find some evidence that suggsts that the value of a transactive memory is curvilinear with the size of the group. They found that if the group is fairly small, the difference in speed to completion of a task by the agents was about the same regardless of whether the group had a transactive memory or not. There was a definite benefit of TMS when groups were larger (between 15 and 27), but the benefit reduced for larger groups (35). I personally think that this is an artifact of how the agent's task is structured, but it does seem fairly reasonable. The last simulation suggested that transactive memory is particularly useful if the group completes multiple different kinds of tasks that are completed in alternating order. A transactive memory allows the groups to more quickly shift tasks, leading to a consistency in time to completion.
Though the studies in this report were not all successful, I found it particularly interesting. The ability to try out new ideas that this study provided also certainly helped the researchers develop their later studies and directed other researchers toward these topics.
I think this post completes my sequence on cognitive interdependence for now, though I'm sure it will crop back up :P
Monday, April 7, 2014
Coginitive Interdependence [Deep Dive] - Moreland series - Part 2
The US Army funded work by four social scientists in Pittsburgh all centered around the influence of turnover on small groups. Work groups in the Army often experience member turnover for a variety of reasons (e.g. transfer, injury, death, etc.) which makes their interest in this area very understandable. In this post, I hope to walk through some of the studies that the Army funded. As far as I know, only one of the studies in this set has been published in an academic journal. There was, however, a technical report given to the Army that I will be basing the information from this post on. This report can be found here: http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA433897
On the project were 4 primary researchers: John Levine, Dick Moreland, Linda Argote, and Kathleen Carley. Dick and Linda were mentioned in prior posts on cognitive interdependence from their extremely important experiments. John Levine was a frequent collaborator with Dick and Linda who is also interested in group behavior. Kathleen Carley is a somewhat different kind of researcher, specializing in computational simulations. In computational simulations, researchers create a set of rules for a world and then see what the outcomes of the world are once the actors in the world interact for a while. The rules in the world can then be adjusted to see if the actors behave much differently or if the outcomes are different. From the abstract of the study, we can see that the researchers intended to gain insight into how personnel turnover impacted groups completing different kinds of tasks. The variety of the researchers also allowed the use of laboratory and simulation-based approaches. Due to two of the researcher's prior investment in the concept of transactive memory, this was included as a component in these studies. Indeed, the lab studies that these researchers competed were a direct extension of those studies.
Productivity Experiment 1 - Turnover and Rumors of Turnover
In the first study, groups of 3 were trained together on a construction task (it isn't made completely clear but I believe it was the radio assembly task used in Liang et al., 1995). There were two manipulations: the groups were warned that there would be turnover (or not) and groups experienced turnover (or not). The warning occurred before the group trained together and the turnover occurred at the beginning of the second performance session. The researchers measured transactive memory and two measured of performance: whether the group could recall the task without having access to the circuit and assembly errors. The results for this first study, in the words of the researchers "were difficult to interpret".
When groups didn't actually experience turnover, they recalled more of the task if they were told that they were going to experience turnover. This makes sense because the group members may have tried more to individually memorize how to do the task if they knew that they couldn't rely on each other. For groups that experienced turnover, however, groups that did not expect turnover recalled more of the task than those that did expect turnover. As for errors, if groups experienced turnover, they performed much better, regardless of whether they were warned that there could be turnover. The researchers made a guess that the newcomers may have just tried really hard, which could explain the effects with errors. In future studies, they made sure to limit the newcomers training harder than the other members.
Productivity Experiment 2 - Turnover and Expertise Information
In this study, all groups were trained together on the task. In the control condition, the group was not warned of turnover and there was no turnover. In the second condition, turnover occurred without warning. In the other three conditions, the groups were warned there would be turnover and then given information about the newcomer's skills. The conditions varied on who received the information, just oldtimers, just newcomers, or both. The researchers measured transactive memory and errors.
As expected, groups that didn't experience turnover made fewer errors than those that experienced unexpected turnover. Groups in the other three conditions where someone received information about the newcomer, all made fewer mean errors than the groups that unexpectedly experienced turnover. Groups where the oldtimers received information about the newcomer made the same number of errors as groups that didn't experience turnover. Interestingly, when the information only went to the newcomer or to both newcomers and oldtimers, groups made slightly more errors. The researchers found nearly mirror results for transactive memory. Groups that didn't experience turnover had the highest transactive memory and groups were oldtimers received information had similarly high levels of TMS.
The researchers then shifted into looking at the effects of turnover on innovation. These studies will be considered next.
On the project were 4 primary researchers: John Levine, Dick Moreland, Linda Argote, and Kathleen Carley. Dick and Linda were mentioned in prior posts on cognitive interdependence from their extremely important experiments. John Levine was a frequent collaborator with Dick and Linda who is also interested in group behavior. Kathleen Carley is a somewhat different kind of researcher, specializing in computational simulations. In computational simulations, researchers create a set of rules for a world and then see what the outcomes of the world are once the actors in the world interact for a while. The rules in the world can then be adjusted to see if the actors behave much differently or if the outcomes are different. From the abstract of the study, we can see that the researchers intended to gain insight into how personnel turnover impacted groups completing different kinds of tasks. The variety of the researchers also allowed the use of laboratory and simulation-based approaches. Due to two of the researcher's prior investment in the concept of transactive memory, this was included as a component in these studies. Indeed, the lab studies that these researchers competed were a direct extension of those studies.
Productivity Experiment 1 - Turnover and Rumors of Turnover
In the first study, groups of 3 were trained together on a construction task (it isn't made completely clear but I believe it was the radio assembly task used in Liang et al., 1995). There were two manipulations: the groups were warned that there would be turnover (or not) and groups experienced turnover (or not). The warning occurred before the group trained together and the turnover occurred at the beginning of the second performance session. The researchers measured transactive memory and two measured of performance: whether the group could recall the task without having access to the circuit and assembly errors. The results for this first study, in the words of the researchers "were difficult to interpret".
When groups didn't actually experience turnover, they recalled more of the task if they were told that they were going to experience turnover. This makes sense because the group members may have tried more to individually memorize how to do the task if they knew that they couldn't rely on each other. For groups that experienced turnover, however, groups that did not expect turnover recalled more of the task than those that did expect turnover. As for errors, if groups experienced turnover, they performed much better, regardless of whether they were warned that there could be turnover. The researchers made a guess that the newcomers may have just tried really hard, which could explain the effects with errors. In future studies, they made sure to limit the newcomers training harder than the other members.
Productivity Experiment 2 - Turnover and Expertise Information
In this study, all groups were trained together on the task. In the control condition, the group was not warned of turnover and there was no turnover. In the second condition, turnover occurred without warning. In the other three conditions, the groups were warned there would be turnover and then given information about the newcomer's skills. The conditions varied on who received the information, just oldtimers, just newcomers, or both. The researchers measured transactive memory and errors.
As expected, groups that didn't experience turnover made fewer errors than those that experienced unexpected turnover. Groups in the other three conditions where someone received information about the newcomer, all made fewer mean errors than the groups that unexpectedly experienced turnover. Groups where the oldtimers received information about the newcomer made the same number of errors as groups that didn't experience turnover. Interestingly, when the information only went to the newcomer or to both newcomers and oldtimers, groups made slightly more errors. The researchers found nearly mirror results for transactive memory. Groups that didn't experience turnover had the highest transactive memory and groups were oldtimers received information had similarly high levels of TMS.
The researchers then shifted into looking at the effects of turnover on innovation. These studies will be considered next.
Thursday, April 3, 2014
Coginitive Interdependence [Deep Dive] - Moreland Series Part 1
In this post, I hope to describe in more detail a few of the transactive memory studies that were conducted at the University of Pittsburgh and Carnegie Mellon University. Richard Moreland was typically on these studies with Linda Argote also involved in several. These researchers were continuing the series of studies that began with the seminal paper with Diane Liang as lead author that was published in 1995. This study was followed in 1996 and 1998 by other experiments. It was not until 2000 that another TMS paper by this group was accepted into an academic journal.
Richard Moreland had been involved with the transactive memory studies using the electrical circuit tasks since the beginning. He, like Daniel Wegner, was a social psychologist and was primarily interested in how this interdependent view of memory influenced what was known about group psychology. He, with frequent coauthor John Levine, had been extremely influential in the area of groups research. Dick, as Moreland often goes by, and John had both proposed a fairly comprehensive theory of group socialization throughout the 80s that had been widely accepted. The seminal aspect of this theory in the chart below.
Before and after a member joins the group, their level of commitment increases to the group up to a point. At different points in an individual's commitment, they are likely to be accepted, to put in more effort, and eventually to leave the group. Their work after these theory was, to a certain extent, focused on how group members could be brought up the commitment curve faster and be more quickly socialized. This interest, I believe, led the researchers to consider group training and transactive memory as an interesting avenue to explore.
After the initial round of studies, these researchers felt like they had a good handle on the phenomena of transactive memory development. Group members spending time together led them to have more accurate perceptions of expertise, leading the group members to more easily coordinate and trust one another. The manipulation to encourage transactive memory, however, includes more information than just expertise to the group members. It could possibly lead the group member to like one another more because they have spend more time together. A few of the experiments controlled for this factor but the researchers thought that there might be other ways to methodologically deal with this concern.
Enter, Moreland and Myaskovsky (2000). Wegner's theory and the prior papers proposed that the transactive memory of the group is composed of information about expertise. In Wegner's experiments, this was due to the romantic couple spending time together and in the earlier Moreland studies it was due to the group members interacting during the training period. In the 2000 paper, however, the researchers isolated the manipulation to just the aspect that the theory mentions, information about expertise. I think this study is perfect in that is smartly builds on prior work, isolating the mechanism, but keeps many other aspects identical which allows us to generalize the findings to past work more easily.
In Moreland and Myaskovsky (2000), all of the members engaging in the radio construction task worked independently or in a group during that first meeting. Then, for half of the independent groups, their work was systematically graded based on area of ability and compared to the other members. A member would then receive a sheet that said the rank of each group member on each of several different categories of skill. Other groups did not receive this information. The researchers found that just providing this limited information about other members, groups performed just as well as if they had been trained together. This suggests that the information about the member's relative skill is helpful for performance, and as helpful as training the group members altogether. The groups that received this performance feedback were not statistically different from the groups that trained together in their level of TMS as measured from the video tapes. Granted, the groups that received performance feedback instead of training together did perform worse and have lower TMS at a mean level, but the values were close.
This particular study attracted the US Army's attention. These researchers applied for and received a grant from the Army to more deeply investigate the effects of performance feedback on groups, especially groups that have employee turnover (like many army groups do). The Army was interested if transactive memory is helpful in small work teams and if providing individualized performance feedback to the group could be a way of quickly building a team's sense of being a group and performance. I will next discuss these studies (never formally published but available in a technical report).
**Personal information about the researchers was attained second-hand and may not be accurate.
Richard Moreland had been involved with the transactive memory studies using the electrical circuit tasks since the beginning. He, like Daniel Wegner, was a social psychologist and was primarily interested in how this interdependent view of memory influenced what was known about group psychology. He, with frequent coauthor John Levine, had been extremely influential in the area of groups research. Dick, as Moreland often goes by, and John had both proposed a fairly comprehensive theory of group socialization throughout the 80s that had been widely accepted. The seminal aspect of this theory in the chart below.
Before and after a member joins the group, their level of commitment increases to the group up to a point. At different points in an individual's commitment, they are likely to be accepted, to put in more effort, and eventually to leave the group. Their work after these theory was, to a certain extent, focused on how group members could be brought up the commitment curve faster and be more quickly socialized. This interest, I believe, led the researchers to consider group training and transactive memory as an interesting avenue to explore.
After the initial round of studies, these researchers felt like they had a good handle on the phenomena of transactive memory development. Group members spending time together led them to have more accurate perceptions of expertise, leading the group members to more easily coordinate and trust one another. The manipulation to encourage transactive memory, however, includes more information than just expertise to the group members. It could possibly lead the group member to like one another more because they have spend more time together. A few of the experiments controlled for this factor but the researchers thought that there might be other ways to methodologically deal with this concern.
Enter, Moreland and Myaskovsky (2000). Wegner's theory and the prior papers proposed that the transactive memory of the group is composed of information about expertise. In Wegner's experiments, this was due to the romantic couple spending time together and in the earlier Moreland studies it was due to the group members interacting during the training period. In the 2000 paper, however, the researchers isolated the manipulation to just the aspect that the theory mentions, information about expertise. I think this study is perfect in that is smartly builds on prior work, isolating the mechanism, but keeps many other aspects identical which allows us to generalize the findings to past work more easily.
In Moreland and Myaskovsky (2000), all of the members engaging in the radio construction task worked independently or in a group during that first meeting. Then, for half of the independent groups, their work was systematically graded based on area of ability and compared to the other members. A member would then receive a sheet that said the rank of each group member on each of several different categories of skill. Other groups did not receive this information. The researchers found that just providing this limited information about other members, groups performed just as well as if they had been trained together. This suggests that the information about the member's relative skill is helpful for performance, and as helpful as training the group members altogether. The groups that received this performance feedback were not statistically different from the groups that trained together in their level of TMS as measured from the video tapes. Granted, the groups that received performance feedback instead of training together did perform worse and have lower TMS at a mean level, but the values were close.
This particular study attracted the US Army's attention. These researchers applied for and received a grant from the Army to more deeply investigate the effects of performance feedback on groups, especially groups that have employee turnover (like many army groups do). The Army was interested if transactive memory is helpful in small work teams and if providing individualized performance feedback to the group could be a way of quickly building a team's sense of being a group and performance. I will next discuss these studies (never formally published but available in a technical report).
**Personal information about the researchers was attained second-hand and may not be accurate.
Subscribe to:
Posts (Atom)