This seems sketchy in that they're giving themselves N*M opportunities to find significant correlations, which they acknowledge -- but with a bias that a significant correlation is not expected to be spurious if it aligns with a claim previously made elsewhere.
> Some of the additional statistically significant findings (e.g., the perception of Rubin’s Vase and openness, the Horse-Seal and intuitive decision making, and the Duck-Rabbit and extraversion and conscientiousness) appear somewhat isolated, are not related to previous research, or claims being made in social media posts and websites. As such, they may be the result of multiple analyses.
I know there's a body of work on "False Discovery Rate", and I think it would be more appropriate to use some of those tools to directly adjust for the number of attempts they're giving themselves.
Correct, FDR control procedures have been part of the statistics literature for at least 50 years, and are relatively easy to implement by hand. There's no excuse.
Yes, and it's a legitimate problem in science. In principle (not that anyone does this), you should probably apply some kind of FDR or FWER control procedure to the entire body of literature in each field. And that's before you get into the known effect of publication bias.
> But I’ve noticed the way people respond to ambiguous statements reveals a lot about their internal state
There are well-documented cognitive biases associated with different forms of ambiguous stimulus interpretation.
It's a target of cognitive-behavioral therapy, for instance, as individuals who are depressed or anxious tend to interpret ambiguous stimuli in certain ways (in general; every person is different; for example, interpreting an ambiguous social situation as reflecting disapproval).
Aggressive individuals are more likely to exhibit a hostile attribution bias as well. Specifically, they're more likely to interpret ambiguous social situations as reflecting hostile intent.
Even some projective tests demonstrate validity. The arguments in those cases tend to be about norms and how to score them and interpret them (to your point, I think there's some evidence that a Sentence Completion Test has a bit more validity than some others).
This was kind of an interesting paper but a poor test of the projective/ambiguous stimulus interpretation hypothesis in general. On the other hand, I guess people are positing this stuff so it's important to have empirical boundaries around it.
A lot of the work in this area has kind of died down due to controversies in the past and a stigma around psychodynamic theories. I've always thought it would be interesting to use modern LLM/DL models with ambiguous stimuli to test these hypotheses (and if they work out, develop prediction and scoring methods) in a more rigorous and thorough way. I think there's always been a kind of problem with past research of how to handle the responses, which are basically natural language responses, which are harder to code etc than T/F, rating scale etc. But if you have LLMs, seems like you can make it more tractable.
There is this theory that structured stimuli (e.g., an item on a typical questionnaire or test) allows you to more precisely target a process and act as primes for recognition. That is, if you want to know about X, you're better off asking or probing specifically for X, and that will prime a respondent through recognition, which is more direct than asking indirectly about something which might initiate an off-target process involving more of a recall process, which is less reliable.
That is definitely true, but it seems like sometimes you don't know what you're looking for, or there might be a kind of impression management problem where people are motivated to respond in certain ways. I don't think you can ever get around that completely, but getting at something in multiple ways often seems better than getting at it in just one way.
> But I’ve noticed the way people respond to ambiguous statements reveals a lot about their internal state
Which then opens up more questions–is that moment in time quality of state something of that particular moment, or of that particular context, or a more stable disposition? I could see the image assessment working to fulfill whatever theory is used by the assessor, something similar to fundamental attribution error.
The fundamental attribution error refers to an individual's tendency to
attribute another's actions to their character or personality, while
attributing their behavior to external situational factors outside of
their control.
Anything can provide psychological insight! The primary action is the subject interpreting and responding to something. It doesn't matter what they are engaging with.
I've been watching police interrogation videos, and it's staggering how many people say, unprompted, "It's not like I would just ___", where ___ is exactly what they did and are trying to hide. It's on their mind, so it comes out.
Where it breaks down is when you try to "science" it, and make a blanket rule that applies to all people in all situations.
I wouldn't doubt that the possibility is there using it that way. But it just seems really impractical and inefficient. I think serious efforts at collecting personal data, involving databases and scripting to scrape data from websites that would not otherwise want to freely give it up, compromising passwords and so on. I don't know what actionable data you get from learning that someone judges a particular ink blot to look like a butterfly.
I think the explanation for their prevalence is just that they found their place in the regular churn of memes, jokes, and quizzes on social media.
I don't believe it has anything to do with the results of the test, they probably just toss that out. It's that you grant these companies partial access to your Facebook account when you take these quizzes.
An interesting quality of the images is that they all have a high degree of symmetry, which is only found in living things. So there is a built in bias.
A key point of Rorschach tests is that they are always the same plates, so there is a large body of literature and case studies specific to interpretations of each one. I think that like any sort of psychological inventory the results are extremely context dependent and require a trained professional to interpret. The trending stuff on social media as mentioned in the article strikes me as more akin to the pseudoscientific misappropriation of things like the MBTI to compare different people in a non-clinical setting, and AFAIU there is really no scientific basis for any such thing.
He's my theory: We use backstory to resolve ambiguity.
The ambiguity: "What's this a picture of?"
A possible answer: "a duck"
A backstory that gets us there: "The area colored white is background, and the area colored black is the foreground, and a black area with that shape of edge matches the silhouette of a duck."
Another ambiguity: "Why is the white not foreground?"
A backstory that gets us there: "Paper is white, ink is black, and when I want to draw, I use the ink to draw the duck on the paper."
So can we conclude that any person who answers "duck" got there through this particular backstory? Of course not! There are plenty of alternative paths to this conclusion.
The intention behind any ambiguous expression is lost forever. The choice of backstory will always be arbitrary.
Betteridge's law of headlines applies: Any headline that ends in a question mark can be answered by the word no. (If the answer was yes, that would be the headline instead.)
Lack of empirical support does not imply empirical support of no insight. In fact, it seems like you can reasonably draw whatever conclusion you please with about equivalent (zero) evidence. Calling these "myths" seems like a bit of a stretch—perhaps "popular conception" might be more accurate.
This seems sketchy in that they're giving themselves N*M opportunities to find significant correlations, which they acknowledge -- but with a bias that a significant correlation is not expected to be spurious if it aligns with a claim previously made elsewhere.
> Some of the additional statistically significant findings (e.g., the perception of Rubin’s Vase and openness, the Horse-Seal and intuitive decision making, and the Duck-Rabbit and extraversion and conscientiousness) appear somewhat isolated, are not related to previous research, or claims being made in social media posts and websites. As such, they may be the result of multiple analyses.
I know there's a body of work on "False Discovery Rate", and I think it would be more appropriate to use some of those tools to directly adjust for the number of attempts they're giving themselves.
Correct, FDR control procedures have been part of the statistics literature for at least 50 years, and are relatively easy to implement by hand. There's no excuse.
Is this the green jelly beans xkcd?
Yes, and it's a legitimate problem in science. In principle (not that anyone does this), you should probably apply some kind of FDR or FWER control procedure to the entire body of literature in each field. And that's before you get into the known effect of publication bias.
I can’t speak to images
But I’ve noticed the way people respond to ambiguous statements reveals a lot about their internal state
(Eg do they assume you’re well-intentioned, assume mal-intent, default to status games, etc)
So I could see the image assessment working in theory if not in application
> But I’ve noticed the way people respond to ambiguous statements reveals a lot about their internal state
There are well-documented cognitive biases associated with different forms of ambiguous stimulus interpretation.
It's a target of cognitive-behavioral therapy, for instance, as individuals who are depressed or anxious tend to interpret ambiguous stimuli in certain ways (in general; every person is different; for example, interpreting an ambiguous social situation as reflecting disapproval).
Aggressive individuals are more likely to exhibit a hostile attribution bias as well. Specifically, they're more likely to interpret ambiguous social situations as reflecting hostile intent.
Even some projective tests demonstrate validity. The arguments in those cases tend to be about norms and how to score them and interpret them (to your point, I think there's some evidence that a Sentence Completion Test has a bit more validity than some others).
This was kind of an interesting paper but a poor test of the projective/ambiguous stimulus interpretation hypothesis in general. On the other hand, I guess people are positing this stuff so it's important to have empirical boundaries around it.
A lot of the work in this area has kind of died down due to controversies in the past and a stigma around psychodynamic theories. I've always thought it would be interesting to use modern LLM/DL models with ambiguous stimuli to test these hypotheses (and if they work out, develop prediction and scoring methods) in a more rigorous and thorough way. I think there's always been a kind of problem with past research of how to handle the responses, which are basically natural language responses, which are harder to code etc than T/F, rating scale etc. But if you have LLMs, seems like you can make it more tractable.
There is this theory that structured stimuli (e.g., an item on a typical questionnaire or test) allows you to more precisely target a process and act as primes for recognition. That is, if you want to know about X, you're better off asking or probing specifically for X, and that will prime a respondent through recognition, which is more direct than asking indirectly about something which might initiate an off-target process involving more of a recall process, which is less reliable.
That is definitely true, but it seems like sometimes you don't know what you're looking for, or there might be a kind of impression management problem where people are motivated to respond in certain ways. I don't think you can ever get around that completely, but getting at something in multiple ways often seems better than getting at it in just one way.
> But I’ve noticed the way people respond to ambiguous statements reveals a lot about their internal state
Which then opens up more questions–is that moment in time quality of state something of that particular moment, or of that particular context, or a more stable disposition? I could see the image assessment working to fulfill whatever theory is used by the assessor, something similar to fundamental attribution error.
https://online.hbs.edu/blog/post/the-fundamental-attribution...A fair point
Anything can provide psychological insight! The primary action is the subject interpreting and responding to something. It doesn't matter what they are engaging with.
I've been watching police interrogation videos, and it's staggering how many people say, unprompted, "It's not like I would just ___", where ___ is exactly what they did and are trying to hide. It's on their mind, so it comes out.
Where it breaks down is when you try to "science" it, and make a blanket rule that applies to all people in all situations.
I see these on Facebook and have avoided them. Isn't the whole point of the "tests" to collect personal data?
I don't know if it's "a new type of psychological myth" so much as a scam.
I wouldn't doubt that the possibility is there using it that way. But it just seems really impractical and inefficient. I think serious efforts at collecting personal data, involving databases and scripting to scrape data from websites that would not otherwise want to freely give it up, compromising passwords and so on. I don't know what actionable data you get from learning that someone judges a particular ink blot to look like a butterfly.
I think the explanation for their prevalence is just that they found their place in the regular churn of memes, jokes, and quizzes on social media.
I don't believe it has anything to do with the results of the test, they probably just toss that out. It's that you grant these companies partial access to your Facebook account when you take these quizzes.
https://www.aclu.org/news/national-security/quiz-what-do-fac...
Tests to determine someone's personality are totally useless, yet they are everywhere. What actionable insight could you even gain.
Isn't this the whole point of Rorschach tests? They are controversial, but still have professional support.
An interesting quality of the images is that they all have a high degree of symmetry, which is only found in living things. So there is a built in bias.
A key point of Rorschach tests is that they are always the same plates, so there is a large body of literature and case studies specific to interpretations of each one. I think that like any sort of psychological inventory the results are extremely context dependent and require a trained professional to interpret. The trending stuff on social media as mentioned in the article strikes me as more akin to the pseudoscientific misappropriation of things like the MBTI to compare different people in a non-clinical setting, and AFAIU there is really no scientific basis for any such thing.
He's my theory: We use backstory to resolve ambiguity.
The ambiguity: "What's this a picture of?"
A possible answer: "a duck"
A backstory that gets us there: "The area colored white is background, and the area colored black is the foreground, and a black area with that shape of edge matches the silhouette of a duck."
Another ambiguity: "Why is the white not foreground?"
A backstory that gets us there: "Paper is white, ink is black, and when I want to draw, I use the ink to draw the duck on the paper."
So can we conclude that any person who answers "duck" got there through this particular backstory? Of course not! There are plenty of alternative paths to this conclusion.
The intention behind any ambiguous expression is lost forever. The choice of backstory will always be arbitrary.
Betteridge's law of headlines applies: Any headline that ends in a question mark can be answered by the word no. (If the answer was yes, that would be the headline instead.)
Lack of empirical support does not imply empirical support of no insight. In fact, it seems like you can reasonably draw whatever conclusion you please with about equivalent (zero) evidence. Calling these "myths" seems like a bit of a stretch—perhaps "popular conception" might be more accurate.