If I wasn't convinced before - really, I can't remember - Eric Schwitzgebel has convinced me here, here, and (to a lesser extent) here that naive introspective reports are not a reliable indicator of the contents of subjects' experiences. He has a number of good examples to show this. I want to point out one example which might not be so good, and why I think it probably isn't.
This figure is meant to induce the so-called "horizontal-vertical illusion." Look at the figure closely. Does either line seem longer than the other?
Well, it's not clear. I have a slight tendency to judge that the vertical line seems longer. But since I learned that the vertical line is supposed to appear longer than the horizontal line when I was first shown the figure, that tendency is a little suspect. Does it sort of "seem" that way in the sense that, or because, the suggestion that the vertical line is supposed to seem longer sort of influenced me? Or does it sort of "seem" that way in the sense that, or because, my introspecting on the experience of viewing the figure and assessing the length of the two lines sort of worked?
Two things. First, I think that introspection yields a great wealth of reliable results in this situation. It seems quite clear to me that the vertical line does not seem two millimeters or more longer than the horizontal. Now, it would be interesting if the vertical line did seem longer than the horizontal line - even just a little bit. And if that is the case, I think we would need some highly trained introspectors to reveal that phact (= phenomenal fact) to us. But, if there is a phact of the matter as to whether or not one line seems longer than the other, that is seemingly the only phact in the case of looking at this figure that we can't reliably apprehend via introspection. So, if there is this phact of the matter - yes, one line might seem longer than the other without our being able to tell that it does. But if the point of the example is to show that there are gross features of our experience that are unavailable to naive introspection, I think it fails.
Second is a suspicion of mine. I don't think we should be all that sure that "the vertical line seems longer than the horizontal line to me" really attributes a phenomenal property to me (or some of my mental states). I have this doubt, not because I think "seems" should be understood strictly in terms of some non-phenomenal feature of our mental make-up (e.g. our credence or tendency to believe that the vertical line seems longer than the horizontal), but because I think it is likely that "x seems longer than y to z" does not pick out a real phenomenal property for all values of "x", "y", and "z". Sometimes I think there is a tendency among philosophers to think that for any property of physical objects P, if P is observable to the naked eye, then sentences of the form "it seems to x that P(y)" attribute a real phenomenal property to "x" (or one of x's mental states, if you like). This thought, together with the thought that "is longer than" picks out an observable property, might be behind the presumption (not necessarily Schwitzgebel's) that "the vertical line seems longer than the horizontal line to me" attributes a phenomenal property to me. But (a) I'm not sure why we should believe this general claim about "seems" and observable properties, and (b) even if we should, it's not clear that "is longer than" picks out an observable property. After all, how often are we actually called on to judge, with the unaided eye, that two lines are exactly the same length, or (what is almost the same) that one or the other is even just the smallest bit longer? And, I should add, if "the vertical line seems longer than the horizontal line to me" does not attribute a real phenomenal property to me, then that would seem to explain why using introspection to determine whether that sentence is true doesn't work and leaves me confused.