I just listened to a talk about the potential benefits of computerized psychological assessment. Besides simplifying data managment and scoring, there are some things that can (almost) only be done with a computer, like adaptive testing or reaction time measurement — the overwhelming majority of studies in cognitive psychology depends on reaction time measures. But I also experienced a slight discomfort in listening to that talk. The speaker gave an example of the kind of research I react allergically to in psychological assessment: He devised a test which he called “visual comprehension” (in German it was actually “Seh-Verständnis” which is not quite identical with “visual comprehension”, but I can’t come up with a better translation). Taking that test, you will be shown short educational videos (4-5 minutes) for the natural sciences; there’s a speaker explaining, and what he is talking about is illustrated by the visual content. Afterwards you will be asked several questions about the content of the video.

So far no problem. But what I don’t like is that this is called “visual comprehension”. Performance in such a task will depend only slightly on visual abilities (as opposed to auditory, say). You can only watch the video once and you won’t be able to re-view portions of the video when you have to answer the questions. So I would say much of the performance depends on memory systems; how much you are able to memorize things while watching, and how good you are at recalling things when confronted with the question. In any textbook on learning and memory you will find that performance in such tasks depends greatly on previous subject knowledge: whenever you already know something about the subject at hand, it will facilitate encoding, because you already know what to look for, and retrieval, which can be explained by associationist models of memory, where specific nodes will be more easily activated the more they are connected to other nodes and the more such connections have been active in the past.

So how much does this test of “visual comprehension” differ from more traditional tests of learning abilities, as for example tests of “reading comprehension”, which have been included in international assessment programmes such as PISA (there called “reading literacy”)? Not much, you might have guessed already, and that is also what the speaker found out. However, it would obviously not be valid to say “visual comprehension is highly correlated with reading comprehension”, because it is just stupid to talk of “visual comprehension” for the kind of test used. But it is a general problem in psychological assessment that people design a test, then give it a name, and then make claims such as “[insert test name] correlates highly with [insert construct of choice]!”.

Sometimes you really should think before you act, or devise some clever test.

Instead of just making bad jokes about zombie arguments, I have now begun reading David Chalmer’s book “the conscious mind“, and will probably be posting about consciousness and such (Chalmers says there is no really good definition, and that seems to be too true). One thing that nags me about the zombie argument becomes already apparent in the introduction to the book: Chalmers states that it is intuitive that there is no need for phenomenal experience; that means he says even if we “feel” something when e.g. we sit at our computers and write blog posts it wouldn’t be necessary that we feel something. He seems to claim (I am still in the first chapter) that all we can do as human beings (thinking, acting) could also be done without experiencing anything. That is the first premise of the zombie argument: That there could be beings just like humans in all respects with the exception that these beings, the zombies, don’t feel anything.

I do not see how this idea should be intuitive. Actually, I think it is really quite contrived. Maybe, if one really tries hard, one might think of a robot that can react to its environment and maybe even initiate actions itself without meaningful experiences. But as human beings, we act and react in most instances particularly because we have inner experiences; that is, I go to a concert because I want to feel good at that concert.

Chalmers (I adress this to Chalmers, but there are other zombists) would answer that there would still be no need for subjective experience. Feeling good might be just some kind of biological¹ reward function: i.e. he might state thateven if something is in some way “good”, say biologically or physically, so that some biological function might have been gotten installed during evolution, it would not be necessary that this function is connected with the subjective experience of feeling good.

I don’t find this very compelling, even if I don’t have a strong argument against it yet. Why should something that is good not feel good? Why should phenomenal experience not be produced by or even identical to the mechanisms for the reward function? I do not find it intuitive to dissociate subjective experience from biological function. Why should it not be that subjective experience itself is a biological function? After all, some kind of conscious experience seems necessary at least for some things we humans are able to do. When you try to remember something, say your grandmother, most people would say they are able to produce an inner image of how their grandmother looks like, that is, they will say that they can somehow see their grandmother even if she is not physically around; of course such an imagination will not be the same as the real seeing of the grandmother. And even more to the point, the whole act of imagining a zombie’s characteristics needs some mental operations. How could it be that these mental operations should not be there in some way? How could it be that a human being should not be aware of her or his mental operations? Chalmers seems to postulate that there is something in addition going on to the mental operations. That is in no way intuitive.

Why am I going on so much about intuitiveness? Chalmers puts a lot of emphasis on the point that zombies, physically¹ like us in every respect, are intuitively conceivable. As yet it seems to me that conceivability is just a way of saying something is not strongly counterintuitive, and the zombie argument rests on the assumption of conceivability. There is nothing of empirical evidence in the zombie discussion. The whole argument is, to say it derisively, out of the armchair. I am however not against armchair argumentation. But the argument better be not only intuitive, but also comprehensible. I don’t really buy an argument when somebody can not even explain the premises. To repeat: I have only just started reading Chalmers works, maybe I am not getting things right. So I am very curious about the next chapters.

¹ Daniel Dennett has noted that it is especially the zombists and other dualists, i.e. people that think consciousness can not be explained in terms of the body and brain, who talk about physical laws instead of taking into account that biology, neuroanatomy and -physiology have something to add beyond “pure” physics.

… statistical graphs quality enforcement agency.

See also here.

www.qwantz.com is the definite source for scientists. Ryan North has already added to the debates on methodology, and today he put up another prudent remark on the mind-body-problem.

Talking about zombies, I had my own stab at the issue.

I wonder if David Chalmers has some dinosaurs in his secret below-the-sea chambers.

When I submitted my first paper to a peer-reviewed journal, I was all anxious not only about the quality of the empirical methods and results, but also about language and style, even more since I am not a native speaker. Granted — being a non-native speaker actually might be to my credit since reviewers will probably be less rigorous once they realize where I come from. But believe me, I would take great pains with the stylistic quality of a paper just as well if I were to submit it to a journal published in German (my native tongue).

Now that I have been asked to review other’s manuscripts for scientific journals, I have to realize that other authors don’t always seem to be as thorough and diligent as I thought everyone in the scientific community would be. Before I submit a paper, I carefully go through it several times and I give it to colleagues or friends, in order to at least exclude the most obvious misspellings and wrong grammar, but I also ask the others to check the consistency and comprehensibility of the text. Today I was asked to review a short paper for a German journal in educational psychology, and that paper surely can’t have gone to an internal review process as described above. There were at least four spelling errors on each page, even one in the title, and above that, whole passages were barely understandable because of inconsistently used terms; for example, they wrote of a “correlation between course observation and competency gains”, which seems to imply that observing a course leads to an increase in competency, when they actually meant the correlation between some characteristic of the course that was observed and competency gains. And that wasn’t the only truly bad paper I had on my desk. In another paper, the authors were not only very careless with spelling and grammar, they also didn’t care to collect new data! Even if it is often beneficial and revealing to re-analyze data, in that case there weren’t any really new conclusions, just a difficult-to-interpret mumble-jumble of “looks as if interesting to pursue in further analyses, but not interesting enough for ourselves, the great re-analyzers of previously collected data”.

One of the people I have published with, a seasoned scientist, told me he has the impression that an increasing number of authors cease to make use of an “internal” revision process (i.e. asking your colleagues, friends, or whoever) before submitting a paper and tend to burden some of the more tedious work (making your paper readable instead of just putting together the results and some refs) on the reviewers. He said the reason might be that people say: “the paper will have to revised anyway, so why bother”. But that is not what I think the review process should be about. I want to think about the scientific quality of a paper: does that paper in any way advance our knowledge, our insights, our “Erkenntnisse” about the phenomena analyzed? have the authors proceeded in an acceptable way? Did they adhere to the methodolological standards? I do not want to think about how a paper I haven’t written myself can be made more readable and comprehensible.

I just read John Kihlstrom’s article “The Automaticity Juggernaut” (TAJ), where he gives what he had already handed out to Daniel Wegner: “a good scolding”, in Wegner’s words, when Kihlstrom had commented on Wegner’s “précis of The illusion of conscious will“. In TAJ, Kihlstrom shows that he not only disdains Wegner’s take on the psychological side of mental causation: he extends that view to other prominent social psychologists, most noteably John Bargh. Like that of his opponents, especially Wegner’s, Kihlstrom’s writing is entertaining and provocative, and all have put some of their papers online; if you are interested in psychologists’ views on free will and mental causation (and these days everybody seems to be concerned with the issue), you might want to check.

Now what is this all about?

Conscious will … is an indication that we think we have caused an action, not a revelation of the causal sequence by which the action was produced. (Wegner, p. 649; emphasis original. see link above)

and

Much of social life is experienced through mental processes that are not intended and about which one is fairly oblivious. (Bargh & Williams, 2006, p. 1).

Kihlstrom interprets quotes such as these (and, of course, the longer articles they are taken out of) like statements of the belief that there is no free will. Wegner and Bargh, in Kihlstrom’s interpretation, deny the possibility of conscious mental causation: that what we believe to be the cause of our doing something, i.e. conscious decisions to do something, is the “true” cause of our doing.

Conscious mental causation and automaticity

That puts W, B and K(ihlstrom) among the mainstream of current discussants of free will and such: Like some neuroscientists, W&B seem to believe that consciousness is qualitatively different from other brain processes not only phenomenologically (i.e. we are aware of conscious processes — that’s the whole point about consciousness — but we are not aware of the other brain processes) but also in ontological status: whereas “unconscious” processes can be causes of actions, conscious process can’t.

However, I think that K misunderstands W&B at least a little. W&B only hint at the possibility that consciousness is an epiphenomenon (a weird byproduct without any real meaning, to put it bluntly) of the “really” causally efficient brain processes. They are only precise about the idea that unconscious, automatic (see below) processes can be causally efficient. In contrast, they are vague about what status they want to give consciousness, and nowhere do they explicitly deny that conscious processes can be causally efficient. I will discuss W&B’s — especially Wegner’s — failure to be precise in this respect below.

It wouldn’t be right to say that W&B are talking about the same thing; still, it is not unfair to put them in the same category as K does in his paper. Both W&B’s theses about conscious and non-conscious causation of actions are based upon the same psychological concept: that our actions are at least influenced (whatever that means — it is taken to be somewhat weaker than caused) by external and internal states and events that we are not aware of. W is especially concerned with the “feeling of doing”; he has put up some smart experiments showing that people can be convinced they are actually doing something (e.g. move and stop a cursor across a computer screen with a mouse) even if they are not the “causers” of that action. That is, in W’s experiments it is shown that people can be mistaken when they attribute actions to their own conscious decisions. B is more concerned with automaticity, that means, with psychological processes that are characterized by four properties:

Inevitable evocation: Automatic processes are inevitably engaged by the appearance of specific environmental stimuli, regardless of the person’s conscious intentions, deployment of attention, or mental set.

Incorrigible completion: Once evoked, they run to completion in a ballistic fashion, regardless of the person’s attempt to control them.

Efficient execution: Automatic processes are effortless, in that they consume no attentional resources.

Parallel processing: Automatic processes do not interfere with, and are not subject to interference by, other ongoing processes – except when they compete with these processes for input or output channels, as in the Stroop effect.

(this is taken from K’s article, see link above).

Automatic processes are usually contrasted with controlled processes, meaning conscious control. This “automatic-controlled” distinction is actually one of the hallmarks of the “cognitive turn” in psychology, when the then-prominent behaviorist dogma (don’t even think about the mental!) was finally mothballed; nowadays dual-process theories abound not only in cognitive psychology, but also in social psychology where W&B have staked their claims. In cognitive psychology, the distinction is primarily based on the above definitions; in social psychology, however, some additions or modifications have been made. People like B refer to automatic processes as such processes that can not be consciously controlled. In a typical experiment, participants are subtly presented with stimuli out of the focus of their attention; i.e. they have to perform a “memory task” where all the to-be-remembered words have to do with the elderly. The real, concealed point of the experiment however is that afterwards people who have been primed with the “elderly stereotype” are going to walk more slowly, and are slightly stooping compared with people having been primed with neutral stimuli. So the point is that in social psychology automaticity often refers to the phenomenon of people being influenced by stimuli or internal processes (such as the stereotype activation) even if they are not aware of those influences.

Automaticity, illusion of mental causation, and free will

Taken together, W&B’s results paint the picture of people’s consciousnesses as epiphenomenal; people fooling themselves into believing they are causing their own actions, when those actions are “in reality” shaped by stimuli and processes outside of their conscious awareness. A diagram taken from Wegner & Wheatly (1999) is especially instructive in this regard:

Note that there is only an arrow going from “unconscious cause of thought” to “thought”, and the arrow from “thought” to “action” is labeled “apparent causal path”; and also that “unconscious cause of thought” and “unconscious cause of action” are not the same thing. So this diagram means that conscious thoughts are not the cause of actions — that is hard-core epiphenomenalism.

B, to be fair to him, nowhere puts forward so bold an assertion, and thus really has to be excluded from the discussion. And he is right in doing so.

Logical flaws in the “apparent mental causation” argument against mental causation

W’s case is more difficult. After having been scolded by the commentators in BBS far more than he himself admits in his author’s reaction, he concedes that thought can (sometimes) cause action. And a rigorous analysis of his experiments and his own interpretations of the result, there can only be one conclusion: fooling people into believing they are the “cause” of their actions when they are not has no consequence whatsoever with regard to the far more general question of whether there can be mental causation at all. It is a very simple logical flaw, a non sequitur: Just because sometimes your action has a different cause from that which you are aware of has no consequence for the thesis that some (other) time you can be totally aware of your reasons and then consciously decide to take an action. So Wegner actually falls into a logical trap. (Bargh is also dangerously close to that gap, but he stays further away than W).

Why does Wegner fool us into believing he does not believe in mental causation?

Taken together Kihlstrom’s scolding seems to be a bit overdone, given that both B&W admit that they don’t really mean what they seem to say about mental causation. On the other hand, at least Wegner really deserves a scolding, if, however, for a slightly different reason: he tries to fool us into believing that he, Wegner, does not believe in mental causation. He talks of “apparent mental causation” and of “the illusion of conscious will, and he draws diagrams as that above. These are blatant assertions. Unless (irony involved!!!) there wasn’t some automatic activation of bluntness involved, he must have asserted these statements on purpose. So why is he so “mystified” at the irate reactions his book and articles have stirred up? Why does he wonder about commentators accusing him of logical errors when he really did commit them? One thing I do not believe is that he can be really that surprised. One thing I do believe is that public discussion of hot topics — and the free will topic is always a good candidate for heated discussion — is sure to influence the commercial success of the books involved. I am not talking about the quality of the experiments involved. It is the interpretations that are problematic. There is already a very low goodness-to-bullshit ratio in the free will debate, to paraphrase Eric Thomson. The diagram about causes of actions, thoughts, and “unconscious causes” is not only logically flawed. It is bad science: a prominent scientist must know better than to commit simple logical flaws.

(Warning: serious, difficult content interspersed with ranting and exasperation)

A possible source of empiricism (i.e. overreliance on empirical results combined with devaluation of, or at least actively ignoring, theoretical analysis) in psychology is a phenomenon that often occurs in statistical analyses, but even more so, the more variables you put into the analysis: you won’t get a clear, unambiguous result. Got your attention? Read on!

That was kind of a presumptuous statement; there are (probably many) situations where statistical analysis yields results that are clear and don’t leave much room for interpretation, such as when you compare different medical treatments in a carefully controlled double-blind study and you have a precisely defined outcome. But this setting is like empirical science paradise. In reality, at least for psychologists and other behavioral scientists, problems often start with the impossibility of double-blinding: a therapist knows what kind of intervention he is performing, and only rarely will it be possible to have therapists executing some other kind of intervention than the one they are used to and which they prefer. The consequence is that you can not separate intervention effects from therapist effects.

You might argue that there is at least the broad domain of experimental psychology, where, by definition, experiments are carefully controlled. But what about matters of operationalization? For a well known and excessively used experimental task such as the Stroop-Color-Word task (color words printed in a different color, such as green, and you have to name the print color, not the written word) researchers don’t really agree whether the performance measures “inhibition” or “interference”. Still, I have to say that this is not really a statistical problem.

The real statistical problems turn up once you don’t only compare one group with another, but several groups. Now what to say when you find out two experimental groups differ from the control group, but not from each other? And even getting this kind of result isn’t that easy: there are many procedures for such multiple comparisons, and unfortunately they won’t all tell you the same. And it’s not only the different procedures (and there never is one that can be preferred against the others…), there is the power issue: Would you have gotten a different result if you had included more observations? If you make too few observations, you don’t have enough statistical power — you won’t be able to defend your results against the Chance Explanation: “the results might only be due to a few odd cases”. With enough power, you can make a claim such as: “it is highly improbable that the difference between experimental and control groups is only due to a few odd cases”; in stats-speak this is termed “significance”. Unfortunately, there is also something like over-powering, i.e. examining too many observations: in such a case you might get a statistically significant result that doesn’t mean much, i.e. the difference is tiny.

For two-group analyses, such as in comparing medication against placebo, and for some several group designs, there exists something like an optimal number of observations, one you can calculate beforehand so you won’t get into big trouble with power problems. However, even for this limited number of designs you must also know beforehand what kind of effects you expect, as in “the difference between medication and treatment should be at least so-and-so”. Even if you are able to make such a prediction — which is far from being easy —- you will be left alone whenever you not only compare groups on a single dimension, such as experimental vs. control, but include another factor you want to make comparisons on, e.g. evaluation of the treatment effects for different age groups, gender, etc.

For such complicated designs, which are nonetheless more of a rule than the exception in the behavioral sciences, there does not exist a unique power calculation, given that you could want to analyze several different kinds of comparisons like interaction effects (“the treatment generally is effective, with the exception of females above 50 years”): the optimal number of observations for an interaction effect will be different from that of simple group comparisons. But in most of the applications of such designs, people will want to analyze both kinds of comparisons. And oh, I forgot: most of the power calculations available have some serious requirements, such as equal group sizes, equal variances, and so on — requirements that will be routinely violated in “real-life” research. (for general information about power analysis, there are countless introductory textbooks and web-pages. The problems when doing research instead of just talking about it in a statistics textbook you will be left alone with…).

Leaving experimental designs, things get even worse. Scott Maxwell, himself author of a really good textbook on experimental designs, had two related articles (I based part of this rant on those…) in the prestigiuous journal Psychological Methods in 2000 and 2004, where he first showed the difficulties of sample size calculations for regression analysis (i.e. you don’t have different groups to compare, but instead collect metric measurements for all observed individuals on different variables and relate them to an outcome variable), which are even larger than for multi-factor group comparisons as described above. More important for the point I want to make is that he then argued that a really large number of psychological studies is actually underpowered, but people don’t do anything about it. Why is that so? Because in multi-comparison designs or regression analyses with several variables, chances are high that you will get at least one significant result for the many different comparisons possible. In the 2004 paper, Maxwell showed that for an experimental design with 2 factors, each with 2 levels (e.g. experimental vs. control and male vs. female), with all possible effects of a medium size chances are 71% that at least one of the effects will be statistically significant with only 10 observations per cell of the design (i.e. 10 males in the control group, 10 males in the experimental group,…). So the researcher might contentedly stop collecting data with such a small sample size because he has found a significant effect — thus failing to observe that all possible effects were really there which he would have found out with just a little more effort. For regression analyses things might even get worse; assume you wanted to predict academic success from several predictor variables such as academic competence, appearance, social competence etc. Even if “in reality”, that is, if you were able to examine all possible subjects, all predictor variables are related to academic success, whenever you observe only a small sample, chances are very high that you will find only one significant predictor and then you stop collecting data.

I want to call this effect “statistical self-immunization”, because it prevents people from doing really thorough analyses: even without proper design, chances are very high that there will be at least one significant and thus publishable result whenever you observe many variables on a limited number of subjects. People would only start thinking in really novel ways if they keep on failing. The erratic, one-significant-in-many-possible result however immunizes against thinking outside the box.

Now this was all about serious kinds of analysis strategies. In factor analysis, you are in statistics hell. Exploratory factor analysis can be interpreted any way you like, depending on how many factors you want to find — and even if you use some objectively-seeming criterion, chances are high you won’t find the true number of factors. And in “confirmatory” factor analysis, you have two paradoxically operating power problems: increasing the number of observations high enough for using sophisticated estimation techniques, the model-that-should-be routinely does not fit because of overpowering (and other problems, but that’s a different story). So then people turn to all kinds of cheating “minor modifications“, “item parceling“, “correlated disturbances“; they will ignore model equivalence, and they will turn to “multifaceted conceptions of model fit” in order to find the one conception that serves their purpose (publication).

And if you still can’t find a fitting model, don’t despair. There’s always partial least squares for structural equation modeling …

Psychology is (at least in the mainstream) an empirical science. What counts in the scientific psychological community and for most of the psychologists that have been educated scientifically are the results of empirical research. For example, many psychologists will sneer at psychoanalytic theory; however, they are much less likely to sneer at the results of psychoanalytic therapy, given that it is notoriously hard to demonstrate substantial differences in effectivity between the more established forms of therapy. [Note: I am not an expert in psychotherapy; there are differences in effectivity conditional on the kind of disorder; there are difference in effectivity conditional on the person to be therapized].

Another example, going further into basic research, is a study on executive functions by Miyake et al. (2000). Executive functions are those that enable us to maintain our acting on a goal set over time and to organize different kinds of behavior required to reach a goal; e.g. to shift our attention between the paper we have to write and the students that keep knocking at our door, or to refrain from ordering a pizza and instead stick to the salad. With their factor-analytic study of the empirical relations between different tasks presumed to measure executive functions, Miyake et al, (2000) single-handedly took over definitional authority of what counts as executive functions at least in the land of experimental psychologists, exemplified by the more than 600 citations their paper brings up in google scholar. Now, many experimental psychologists would agree that executive functions are comprised of “shifting, updating [working memory] and inhibition”; see e.g. a recent review on executive functions in preschool children. No theoretical classification of executive functions would ever have been so successful in psychology.

Isn’t that actually a good thing? Of course, empirical results really should count more than theory whenever there is no empirical evidence to support a theory. But the condition in italics is important. To forgo theory can seriously stall empirical success. My pet example is personality psychology: many psychologists are just so convinced that there are universal traits that can be observed in every single human being, and that there are only differences in degree but not in kind, i.e. every person can be assigned a value in extraversion. If you stay inside that approach and don’t ever think of changing to another perspective, you are going to have to get attuned to weak empirical associations between those trait scores and other criteria. Walter Mischel has famously criticised trait psychology for being unable to generate correlations with relevant criteria above r = .30 (which is not very high) in a book published in 1968, and things have not changed very much in the last 40 years — see e.g. Barrick, Mount and Judge’s meta-meta analysis on the relation of the Big Five to job performance.

Why researchers still hand on to their old theories and methods is somewhat mysterious. One reason might be that the results of many empirical-statistical methods are not unequivocal. For example, the disappointingly low correlations of personality trait measures with criteria such as job performance are often explained away with low reliabilities and range restriction. In essence this leads to conclusions like “if the measures had perfect reliability and if there was no range restriction, the ‘real’ correlation would be much higher than the one observed”. So instead of thinking about better ways to measure something or contemplating a theoretical change, people stick to their old suboptimal ways.

Just yesterday I wrote about the lexical hypothesis in personality psychology. Even if there are now some researchers proposing a six-factor general personality trait theory, the Big Five (or the OCEAN model: openness, conscientiousness, extraversion, agreeableness, and neuroticism) is probably the most popular general personality (trait) theory by far. Among the Big Five theorists, there is a group led by Paul Costa and Robert McCrae who have somewhat dissociated from the lexical origin of the Big Five and in their more recent writings propose that the Big Five are not only descriptive terms, but biologically based “basic tendencies” (e.g. in McCrae, Costa, Ostendorf, Angleitner, Hrebickova et al., 2000)¹. However, they do not really explain what “basic tendencies” might mean; instead they put forward evidence for cultural generalizability and long-term stability mainly of questionnaire scores. The argumentation is kind of backwards: They say, “look, the questionnaire scores are cross-culturally and intraindividually stable. They must be biologically based”. This is in itself a very weak argument; one might put forward all kinds of objections like that the long-term stability does not really tell us about something to be biologically based, and the cross-cultural generalizability has been heavily disputed, see e.g. here.

Lisa Pytlik Zillig, Hemenover and Dienstbier (2002) have explored another line of evidence against the “basic tendencies” claim of Costa-McCrae theorists. Instead of meeting them on their own grounds, Pytlik Zillig et al. simply took several Big Five inventories (questionnaires and adjective lists) and analyzed the content of the items of these inventories. They tried to classify the items into one of three categories: how much does an item describe affects, behaviors, and cognitions? (the distinction of affect, behavior, and cognition is quite common in psychology; e.g. there are some models of attitudes that assume an attitude has something of all three, an [evaluative] affective component, a behavioral component [how does one react], and a cognitive, non-evaluative component).

The results are somewhat disappointing for the “basic tendencies” idea: across different inventories and across different groups of raters, items for the Big Five factors differ systematically with regard to how much they reflect affect, behavior, and cognition. Most striking is the difference between items assumed to assess neuroticism vs. items for conscientiousness: where items for the former are mainly (60-90%) about affects, items for the latter are mainly about behaviors (again 60-90%), and almost none of conscientiousness items describe affects. Only for items about agreeableness the three categories are represented almost equally.

What does that mean for the idea about the Big Five being “biologically based basic tendencies”? The results cast serious doubt on that idea. If the basic tendencies idea was right, one would assume that all of the Big Five factors correspond to affects, behaviors or behavioral tendencies, and cognitions equally. The result that some factors have more to do with affects (i.e. neuroticism, and to some extent extraversion), others are mainly about cognitions (openness) and a third category has to do with behavior (conscientiousness and extraversion) is much more compatible with the idea that the Big Five are about different things and not operating at the same level, i.e. that they are not (all) basic tendencies. If one thinks of a “basic tendency” as something in the brain, say, somebody with a higher level of neuroticism being more excitable in some neural network; how might this “basic tendency” be comparable to a “basic tendency” like conscientiousness that seems to be mainly about behaviors? In the words of Pytlik Zillig et al.:

Assume for the moment that there is some very basic core or reality to Big 5–level traits, that the ABC [affect, behavior, and cognition] dimensions are highly meaningful constructs for assessing that core, and that the operational definitions of traits on ABC dimensions in major inventories reasonably reflect those underlying latent traits. Given those assumptions, our findings suggest that abstract arguments (and conceptual definitions of traits, such as found in personality texts) about the basic nature of traits may miss the mark.

As a final remark: this study again shows that words are not equal to their meanings in personality psychology. The Costa and McCrae theorists claim that their Big Five are basic tendencies, biologically based and causally operating. But their instruments do not support this claim. The content analysis of the instruments reveal that the Big Five are different with regard to what they refer to.

¹ I realize that most of the links require subscriptions, but I can’t help citing peer reviewed sources.