Allowing yourself to be wrong allows you to be right…eventually

The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge.

– Stephen Hawking

It is wise to remember that you are one of those who can be fooled some of the time.

– Laurence J. Peter

We human beings are so good at pattern recognition that sometimes we find patterns that are not even there. I have never seen a cognitive profile, no matter how unusual and outlandish, that did not inspire a vivid interpretation that explained EVERYTHING about a child. In fact, the more outlandish, the better. On a few occasions, some of the anomalous scores that inspired the vivid interpretations turned out to be anomalous due to scoring errors. In these humbling experiences, I have learned something important. I noticed that in those cases, my interpretations seemed just as plausible to me as any other. If anything, I was more engaged with them because they were so interesting. Of course, there is nothing wrong with making sense of data and there is nothing wrong with doing so with a little creativity. Let your imagination soar! The danger is in taking yourself too seriously.

The scientific method is a system that saves us from our tendencies not to ask the hard questions after we have convinced ourselves of something. Put succinctly, the scientific method consists of not trusting any explanation until it survives your best efforts to kill it. There is much to be gained in reserving some time to imagine all the ways in which your interpretation might be wrong. The price of freedom is responsibility. The price of divergent thinking is prudence. It is better to be right in the end than to be right right now.

This post is an excerpt from:

Schneider, W. J. (2013). Principles of assessment of aptitude and achievement. In D. Saklofske, C. Reynolds, & V. Schwean (Eds.), Oxford handbook of psychological assessment of children and adolescents (pp. 286–330). New York: Oxford.

What if we took our models seriously? Slides from my NASP 2014 talk

WISC-IV Five-Factor Model

I was part of a symposium last week at NASP on the factor structure of the WISC-IV and WAIS-IV. It was organized and moderated by Renée Tobin, who edited the special issue of the Journal of Psychoeducational Assessment on the same topic. The other presenters were Larry Weiss, Tim Keith, Gary Canivez, Joe Kush, Dawn Flanagan, and Vinny Alfonso.

The slides for my talk are here. The text for the talk is written in the “Notes” for each slide.

I have spoken at length on this topic in my article on the special issue and in a companion video.

The spreadsheets that accompany the article are here:

WISC-IV Spreadsheet

WAIS-IV Spreadsheet

Time lavished on hypothesis fishing trips is stolen from children we no longer have time to help.

We do what we must, and call it by the best names we can.

– Ralph Waldo Emerson

Cheetahs are the fastest animals on land but do not always catch their prey. For about 60 seconds or so, cheetahs give it their all. After that, they give up. Why? For a cheetah to persist, the expected rewards must justify the caloric expenditure, the risk of injury, and the considerable strain on their bodies that sprinting inevitably causes. In the wild, there is no glory in Pyrrhic victories. Sometimes it is better to cut your losses, even though you could “succeed” with more effort.

There is something analogous that happens in cognitive assessment. For a time, it is worthwhile to vigorously pursue a hypothesis, to clarify an anomalous finding, or to explain a curious behavior. However, when answers are not forthcoming, there is a point at which it is wise to give up, even before all alternatives have been exhausted. The time saved can be devoted to other questions about the child that may be important. It is perfectly acceptable to write in reports that, given the available data, it is not yet possible to distinguish between alternative hypotheses about a child. It is perfectly acceptable to speculate about those hypotheses, provided that those speculations are clearly labeled as such and that it is explicitly stated that the true explanation might not be included in the list of speculations.

This post is an excerpt from:

Schneider, W. J. (2013). Principles of assessment of aptitude and achievement. In D. Saklofske, C. Reynolds, & V. Schwean (Eds.), Oxford handbook of psychological assessment of children and adolescents (pp. 286–330). New York: Oxford University Press.

Dr. Procrustes does not need to see you; he has your test scores.

I rock at the Tower of Hanoi—you could give me a stack of as many discs as you like and I can move the whole stack from one peg to the other without any hesitation and without a single error. I don’t mean to be immodest about it, but it’s true. My performance is like 11.8 standard deviations above the mean, which by my calculations is so rare that if a million people were born every second ever since the Big Bang, there is still only a 2.7% chance that I would have been born by now—I feel very lucky (and honored) to be here.

You would be forgiven for thinking that I had excellent planning ability…but not if you voiced such an opinion out loud, within earshot of my wife, causing her to die of laughter—I would miss her very much. No, it is not by preternatural planning ability that I compete with only the gods in Tower of Hanoi tournaments-in-the-sky. In fact, the first time I tried it, my score was not particularly good. I am not going say what it was but the manual said that I ranked somewhere between the average Darwin Award winner and the person who invented English spelling rules. After giving the test some thought, however, I realized that each movement of the discs is mechanically determined by a simple rule. I will not say what the rule is for fear of compromising the validity of the test for more people. The rule is not so simple that you would figure it out while taking the test for the first time, but it is simple enough that once you learn it, you will be surprised how easy the test becomes.

All kidding aside, it is important for the clinician to be mindful of the process by which a child performs well or poorly on a test. For me, the Tower of Hanoi does not measure planning. For others, it might. Edith Kaplan (1988) was extremely creative in her methods of investigating how people performed on cognitive tests. Kaplan-inspired tools such as the WISC-IV Integrated provide more formal methods of assessing strategy use. However, careful observations and even simply asking children how they approached a task (after the tests have been administered according to standard procedures) is often enlightening and can save time during the follow-up testing phase. For example, I once read about an otherwise low-performing boy who scored very well on the WISC-IV Block Design subtest. When asked how he did so well on it, he said that he had the test at home and that he practiced it often. The clinician doubted this very much but his story turned out to be true! His mother was an employee at a university and saw someone from the Psychology Department throwing outdated WISC-III test kits into the garbage. She was intrigued and took one home for her children to play with.

I once gave the WAIS-III to a woman who responded to the WAIS-III Vocabulary subtest as if it were a free association test. I tried to use standard procedures to encourage her to give definitions to words but the standard prompts (“Tell me more”) just made it worse. Finally, I broke with protocol and said, “These are fabulous answers and I like your creativity. However, I think I did not explain myself very well. If you were to look up this word in the dictionary, what might it say about what the word means?” In the report I noted the break with protocol but I believe that the score she earned was much more reflective of her Lexical Knowledge than would have been the case had I followed procedures more strictly. I do not wish to be misunderstood, however; I never deviate from standard procedures except when I must. Even then, I conduct additional follow-up testing to make sure that the scores are correct.

This post is an excerpt from:

Schneider, W. J. (2013). Principles of assessment of aptitude and achievement. In D. Saklofske, C. Reynolds, & V. Schwean (Eds.), Oxford handbook of psychological assessment of children and adolescents (pp. 286–330). New York: Oxford University Press.

WAIS-IV & WIAT-III Multidimensional Scaling in 3D

Here is a static image of a multidimensional scaling of the WAIS-IV and WIAT-III. It is interactive when you click it (Works best in Firefox & Chrome, not in IE or Safari).

Interactive 3D Multidimensional Scaling of the the WISC-IV

Following the same procedures as I did with the WJ III, I have made an interactive 3D MDS of the WISC-IV. The image below is static but if you click it, you can rotate and zoom the image interactively.

WISC-IV MDS

Crystallized Intelligence, Visualized

Attention Tests & ADHD: A Modest Proposal

Some people did not like my post suggesting that attention tests are really important for understanding a person but that they are not awesome in every way that a test can be awesome.

I’ve found their arguments to be persuasive and now have an updated position:

It Isn’t “ADHD” Unless You Fail My Tests

ADHD is a problem. Is it even real?

If it is real, who has it? For really real, I mean.

What doesn’t work:

1. We cannot trust self-reports, for sure. Kids don’t know anything and adults—well, Ritalin junkies will say or do anything.
2. Parent and teacher rating scales? Don’t get me started.
3. Interviews? Unstandardized rating scales in conversation form. A lotta yada yada for a lotta nada nada.
4. Comprehensive reviews of school records? Are there any studies with real data proving that this works?
5. Direct observation? That’s for grad students and research studies! Who has the time? I’ll “observe” during testing.
6. A comprehensive consideration of all the available evidence, including attention tests? Well, this is the obvious answer people give. It does seem like a reasonable answer but in reality it is not. Too expensive, too much work, and too much “peace, love, & rainbows” if you know what I mean.

What does work:

We need objectivity and we need it quick! Like brain scans or something—but less expensive. Like tests, especially computerized ones with T-scores rounded to 2 decimals of precision.

The good news is that the tests we already have are great. And there are so many of them! If a person performs well on one test, there is always another one you can give. There is a low score in that profile somewhere…

When you find a low score, give the test again for the sake of reliability. Or an alternate form. If the score isn’t low the second time around, keep looking. Usually you won’t have to look long—it’s a rare person who does well on everything. If the person is really smart, count an average score as a deficit.

Whatever the low score is, as long as seems like it measures attention, it is probably the explanation you are looking for. Consult the research literature….Oh good! There was a study once that showed that this test was correlated with…problems (p < 0.05).

Don’t listen to people who claim that our knowledge about ADHD is limited. Ever gone to a library? Those things are freaking huge! And there are lots of them. In universities, even. In one of those books or in one of those journals, somebody with a Ph.D. said something about something (more p-values) that totally agrees with you. Find it. Cite it. There might be more than one study. The Internet can help. Don’t tell me that knowledge is limited until you’ve read everything.

Talking with Evaluees

Now that the explanation has been found, explain to the person that his or her attention problems are really a manifestation of something deeper: executive function deficits. Explain that executive functions are like attention but fancier. Because working memory. Point to a picture of the frontal lobes. There are lots of studies about the brain (p < 0.05).

The goal here is to be right. And you can always be right. You sacrificed your 20s for that degree…you earned the privilege never to be questioned.

Speaking of being questioned, what’s with these people who claim to have attention problems and then reject my non-diagnosis of ADHD because of good test performance? Liars all! Or whiners. Or both. Lying whiners are the worst! I tell them:

“Look, you are completely normal. I’ve seen people with problems way worse than yours. They failed my tests. They have ADHD. You don’t. If you did, you’d’ve have failed my tests. I know, I know—your life is a mess, your work history is spotty, and that you have a long history of failed relationships. There’s no need to call this ADHD. Maybe you should try harder to pay attention to things and do them right—like you did on my tests. Ever think of that? It’s a shame because you’re a bright guy. I’m not just saying that to be nice—you did really well on my tests. You should’ve finished your college degree—nevermind…water under the bridge! I see you’ve been in a number of car accidents. Be safe! Also, return your library books on time—you’re losing a lot of money that way. Get more sleep. Don’t procrastinate any more either—that’s half your problem right there. The amount of time that you play video games is not healthy. All things in moderation as they say! I don’t mean to be rude but I’d like to give you some constructive criticism: When I am trying to say something, you often interrupt with things you are thinking about. Sometimes it isn’t even relevant to what I was saying. I do want to hear you but I did find it a little annoying. I bet other people don’t appreciate it either. A little courtesy goes a long way…I’m just being honest. One more thing, when your wife was here, she said that you mean well but often fail to follow though with things and that she can’t trust you with anything time-sensitive. She seems real nice but if you keep letting her down like that…you are going to lose her. I thought you should know. Keep your promises and things should get a lot better. Okay, now…on your way. Good luck!”

Psychometrics from the Ground Up 9: Standard Scores and Why We Need Them

In this video tutorial, I explain why we have standard scores, why there are so many different kinds of standard scores, and how to convert between any two types of standard scores.

Here is my Excel spreadsheet that converts any type of standard score to any other type.

Overture

When we are at our best, clinicians skilled in psychological assessment see beyond the obvious and communicate something useful and true to a person who needs our help.  Uncovering something true often requires a bit of science, sometimes a little math, and always a lot of empathy. Communicating what is true so that it is useful requires still more empathy and a bit of art. We facilitate in our clients a deeper understanding of what is happening to them and present to them a different vision of what they can be.

In this blog, I hope to communicate something useful and true to other professionals about psychological assessment. I hope that in communicating what I know, I will come to know more than I do now.

The title of the blog is more of an aspiration than a description.