Elsevier

Consciousness and Cognition

Volume 19, Issue 4, December 2010, Pages 1069-1078
Consciousness and Cognition

Measuring consciousness: Is one measure better than the other?

https://doi.org/10.1016/j.concog.2009.12.013Get rights and content

Abstract

What is the best way of assessing the extent to which people are aware of a stimulus? Here, using a masked visual identification task, we compared three measures of subjective awareness: The Perceptual Awareness Scale (PAS), through which participants are asked to rate the clarity of their visual experience; confidence ratings (CR), through which participants express their confidence in their identification decisions, and Post-decision wagering (PDW), in which participants place a monetary wager on their decisions. We conducted detailed explorations of the relationships between awareness and identification performance, looking to determine (1) which scale best correlates with performance, and (2) whether we can detect performance in the absence of awareness and how the scales differ from each other in terms of revealing such unconscious processing. Based on these findings we discuss whether perceptual awareness should be considered graded or dichotomous. Results showed that PAS showed a much stronger performance-awareness correlation than either CR or PDW, particularly for low stimulus intensities. In general, all scales indicated above-chance performance when participants claimed not to have seen anything. However, such above-chance performance only showed when we also observed a correlation between awareness and performance. Thus (1) PAS seems to be the most exhaustive measure of awareness, and (2) we find support for above-chance performance in the absence of subjective awareness, but such unconscious knowledge only contributes to performance when we observe conscious knowledge as well. Similarities and differences between scales are discussed in the light of consciousness theories and response strategies.

Introduction

A systematic comparison of measures of subjective awareness is long overdue (see also Dienes and Seth, 2010, Wierzchoń et al., 2009) since such measures are currently widely used in consciousness research (for an overview, see Seth, Dienes, Cleeremans, Overgaard, and Pessoa (2008)). For instance, the search for the neural correlates of consciousness typically involves contrasting brain activation during task performance with and without awareness (Baars, 1988; see e.g. Christensen et al., 2006, Lau and Passingham, 2006; but see Lamme (2006), for a different view). In this paper, we compare three currently popular measures of subjective awareness and assess how well each correlates with performance in a masked identification task. The Perceptual Awareness Scale (PAS; (Ramsøy & Overgaard, 2004)) is a purely introspective measure that requires participants to indicate the clarity of their experience of a stimulus. Confidence ratings (CR; e.g. (Cheesman and Merikle, 1986, Dienes et al., 1995) require participants to indicate their confidence in their decisions. Finally, post-decision wagering (PDW; Persaud, McLeod, & Cowey, 2007) requires participants to place a monetary wager on the accuracy of their decisions (i.e., stimulus identification). All three measures potentially present substantial advantages over other methods aimed at assessing the relationships between awareness and task performance. In particular the measures can be collected almost concurrently with decisions and can thus be correlated with task performance on a trial-by-trial basis, hence addressing Shanks and St. John’s “retrospective assessment” problem (1994). However, it is unclear which method is most sensitive, that is, which method shows the best relationship between task performance and self-reported awareness. Likewise, it is unclear which measure is most exhaustive, that is, which method reveals the most conscious processing (Reingold & Merikle, 1988).

Dienes et al. (1995) have proposed the “zero-correlation criterion” (see also Chan (1992)) and the “guessing criterion” as tests for such conscious and unconscious processing, i.e. how sensitive and exhaustive the measures are. When analyzing using the zero-correlation criterion one looks for correlations between performance (an objective measure) and self-reported awareness or confidence in being correct (a subjective measure) across different conditions of task difficulty (e.g. various stimulus durations). Any positive relationship between performance and awareness suggests the involvement of at least some conscious knowledge in determining performance. However, the involvement of conscious processes does not exclude the involvement of unconscious processes. To examine these, the “guessing criterion” is used. Using this, performance is assessed for those cases where participants claim to be guessing (that is, when they claim to be performing randomly). If participants’ performance is at chance, there is no knowledge contributing to the task, unconscious or otherwise, and subjective and objective thresholds are identical. If, however, participants who claim to be guessing perform above chance, then one would conclude that their performance is based on knowledge they are not aware of possessing, that is, on unconscious knowledge. An important caveat to this reasoning is that above-chance performance can also be the consequence of the test failing to be exhaustive when subjects claim to be guessing, meaning that participants fail to be complete in their report about their conscious contents. Given this state of affairs, the best one can do is to consider that if one scale indicates less unconscious processing than another, then that scale should be taken to be more exhaustive than the others, all else being equal (that is, assuming that there are no differences in the extent to which each scale promotes awareness in and of itself, and in the extent to which the different scales erroneously labels some unconscious knowledge as conscious knowledge). One should thus look for the scale that is simultaneously most sensitive and most exhaustive. In other words, the most promising scale is the one that (a) shows better correlation than others between performance and awareness at different levels of difficulty (the zero-correlation criterion), and (b) shows the least above-chance performance for trials on which participants claim to be guessing (the guessing criterion).

The current study was thus motivated by two simple goals. First, we aimed at determining whether the three measures predict the same relative contribution of conscious and unconscious processing. To this end, we determined the relationship between performance and awareness at different levels of task difficulty. Additionally, we explored the extent to which each scale indicates the same level of above-chance performance in the absence of awareness, if any (for an overview of this debate, see Kouider and Dehaene (2007), or Overgaard and Timmermans, 2009). The second goal was to explore whether perceptual awareness should best be considered as graded or as dichotomous (e.g., Overgaard et al., 2006, Sergent and Dehaene, 2004). Though there are theoretical complications, comparing the three scales in this light should be informative.

In the current experiment, we compare three scales, each of which measures awareness in a different way. Each of these scales has a number of claimed advantages and disadvantages. Even though some of these are difficult to validate empirically, they will be mentioned in the following as they may still influence the evaluation of the scales.

When using PAS, participants report on the quality of their subjective experience directly. PAS was originally created by the participants in an experiment by Ramsøy and Overgaard (2004). In this experiment, participants were asked to describe the quality of their visual experience as they looked at briefly displayed stimuli, using a scale they had created themselves. It was suggested to participants that they start the scale with ‘No experience’ and ended it with ‘A clear image’, but they were free not to follow the suggestion and/or to use any number of categories. All five participants ended up using a 4-point scale with the elements (1) ‘No experience’, (2) ‘Brief glimpse’, (3) ‘Almost clear image’, and (4) ‘Absolutely clear image’. Although the participants differed in their labeling of the categories, they agreed in their definitions of the categories.

PAS can be claimed to be intuitive in that the categories used are created not by an experimenter, but by other research participants evaluating their conscious experience (Ramsøy & Overgaard, 2004). In addition, as it is not related to a participant’s evaluation of how good their answer is (as is post-decision wagering), PAS and other direct measures of conscious experience can easily be used in tasks in which there is no “correct” answer such as the perception of an ambiguous figure or binocular rivalry. Finally, Persaud and colleagues (2007) have argued that participants using numerical confidence ratings may withhold knowledge, as they have no motivation to reveal it. This criticism also applies to PAS.

One claimed advantage of PAS is that the participants are asked directly to provide the information that experimenter is looking for, that is, their conscious experience. Paradoxically, this is also a claimed disadvantage as it depends on how good participants are at reporting their experience. If the participants are reasonably good introspectionists, then asking them to report directly minimizes the risk of confusion or errors that might arise if the participants are asked about something very different, and their conscious experience is inferred from their answer. However, if participants are poor introspectionists, then asking them to report on their experience is associated with a large risk. Thus, because it seems possible to argue both ways, the answer must be obtained empirically, in comparison with other scales.

Confidence ratings (CRs) have been used either with respect to perception itself, in which case participants directly report their confidence in having perceived something (Bernstein and Eriksen, 1965, Cheesman and Merikle, 1984), or with respect to participants’ performance, in which case they report their confidence in having provided a correct answer. In the former case, CRs closely resemble PAS. In the latter case, however, CRs are metacognitive judgments in which participants express the extent to which they are certain that their answer is correct in forced-choice tasks (Cheesman & Merikle, 1986) or in the discrimination tasks typical of implicit learning research (Dienes et al., 1995, Kuhn and Dienes, 2006). Although CRs may be expressed on very different scales, most variations include ‘guessing’ or ‘no confidence’ in the description of the lowest rating. Examples include dichotomous scales such as “guess/know” and “guess/anything else”, as well as gradual scales.

In many ways, CR have the same advantages and disadvantages as scales that ask directly about conscious experience. Nevertheless, in light of the observation that participants may not be good introspectionists, CR may have an advantage, as participants are not asked directly to introspect. A potential challenge with having participants rate their own performance, however, is that while two participants may have a comparable clarity in their experience of a stimulus, they might use different criteria to decide themselves confident.

Post-decision wagering (PDW) is a recently suggested measure of conscious content (Persaud and McLeod, 2008, Persaud et al., 2007). After performing a task, the participants place a wager on having performed the task correctly. The rationale is that the wagers are based on the awareness of the participants, but the participants never need to introspectively report their awareness. According to Persaud and colleagues, they simply perform a task that requires awareness to be completed. For this reason, PDW has been put forward as an “objective” or direct measure. Persaud and colleagues used wagering dichotomously so that participants could place either a low or a high wager at even odds on one of two possibilities. The degree to which a participant maximizes his gains through advantageous wagering (betting high after a correct decision, or low after an incorrect decision) is assumed to be indicative of conscious experience. More importantly however, since the possibility to gain (real or imaginary) money provides participants with a strong incentive to reveal any conscious knowledge they may possess, failure to wager advantageously should reflect absence of awareness in a more exhaustive manner that other measures.

Despite being intuitive as well as potentially exhaustive, post-decision wagering as a method to assess awareness has been questioned from a theoretical point of view. The claim that PDW is a direct measure of awareness has been questioned by Seth (2008) who argues that it is in fact a second-order judgment of the reliability of a first-order experience. Such a “metacognitive comment” does not exhaustively describe the rich phenomenology of conscious experience and metacognitive competences are susceptible to biases (see Persaud, McLeod, and Cowey (2008), for a reply to the critique). In addition, PDW (as applied by Persaud and colleagues) seems to presuppose that conscious experience is dichotomous. If conscious experience is not dichotomous, however, a problem arises in that criterion setting about when to start wagering high may vary significantly between participants, as recently argued by Clifford, Arabzadeh, and Harris (2008). As a consequence, it is impossible to ascertain, on a trial-by-trial basis, whether a participant was conscious or not, as a low wager is not necessarily synonymous with absence of awareness – participants could be reluctant to take a risk even though they have a vague experience of the stimulus. The influence of risk aversion on wagering behavior was indeed recently confirmed empirically by Dienes and Seth (2010).

Additionally, Clifford and colleagues (2008) argue that when wagering is used as it is by Persaud and colleagues, the optimal wagering strategy is always to bet high, as this will give the same outcome if accuracy is at chance, but a higher outcome if accuracy is above chance, whether above-chance accuracy is subtended by unconscious knowledge or not. This behavior was not observed in the previously mentioned experiments, and Schurger and Sher (2008) report that only 2 out of more than 100 tested participants adopted an optimal strategy even when encouraged to wager high. These results seriously question the claim that wagering is intuitive, but it remains to be seen empirically if it is a more substantial issue for wagering scales than it is for the other scales.

It is often preferable to have a measure of subjective experience that does not alter task accuracy. The possibility of monetary profit, however, has been shown to improve task accuracy, whereas this is not the case for CR (cf. Persaud & McLeod, 2008). When used in neuroscientific experiments, this is a problem, particularly if improved accuracy is caused by emotional arousal. For instance, one might imagine that emotional arousal is different for betting high vs. betting low, and this would make it difficult to differentiate neural activity related to awareness from activity related to emotional arousal.

Section snippets

Experiment 1

The main goal of the experiment was to examine if the three measures allow us to draw the same conclusions about the relationship between performance and awareness. To this end, we had participants perform a masked stimulus identification task. After each response, participants had to perform one of three judgments: (1) rate stimulus visibility (PAS), (2) give a confidence rating in their answer, or (3) place a post-decision wager.

Conclusion

Going with the tacit assumption that objective measures should be preferred over subjective (i.e. introspective) ones when studying consciousness, one would expect results from the present experiment to advocate the use of PDW over CR, which again should be preferred over PAS, given that it is the most “subjective” of the three. However, overall, PDW proved to be less sensitive than PAS for stimuli that were hard to identify, with CR taking up an intermediate position. Thus, the present

Acknowledgements

Axel Cleeremans is a Research Director with the F.R.S.–FNRS (Belgium). This work was supported by The MindBridge project, funded by the European Commission under the Sixth Framework Programme, Contract No. 043457 and by Concerted Research Action 06/11-342 titled “Culturally modified organisms: What it means to be human in the age of culture”, financed by the Ministère de la Communauté Française – Direction Générale l’Enseignement non obligatoire et de la Recherche scientifique (Belgium).

References (26)

  • B. Baars

    A cognitive theory of consciousness

    (1988)
  • I.H. Bernstein et al.

    Effects of “subliminal” prompting on paired-associate learning

    Journal of Experimental Research in Personality

    (1965)
  • Chan, C. (1992). Implicit cognitive processes: Theoretical issues and applications in computer systems design....
  • Cited by (328)

    • Distracted by familiarity: Implications of ‘autopilot’ as a default cognitive mode

      2023, Transportation Research Part F: Traffic Psychology and Behaviour
    View all citing articles on Scopus
    1

    Shared first-authorship.

    View full text