Raven’s Matrices have long been a gold standard for psychologists needing to measure general intelligence. But the good ones, the ones scientists like to use, are too expensive for most research projects.
Christopher Chabris, associate professor of psychology at Union College, and David Engel, postdoctoral associate at MIT Sloan School of Management, think the public can help. They recently launched a campaign to crowdsource “the next great IQ test.” The Matrix Reasoning Challenge, created through MIT’s Center for Collective Intelligence with Anita Woolley and Tom Malone, calls on the public to design and submit matrix puzzles – 3×3 grids that asks subjects to complete a pattern by filling in a missing square.
Chabris says they aren’t trying to compete with commercially available tests used for diagnostic or clinical purposes, but rather want to provide a trustworthy and free alternative for scientists. Because these types of puzzles are nonverbal, culturally neutral, and objective, they have wide-ranging applications and are particularly useful when conducting research across various demographics. If this project is successful, a lot more scientists could do a lot more research.
A simple example of a matrix puzzle. Source: Matrix Reasoning Challenge
“Researchers typically don’t have that much money,” Chabris said. “They can’t afford pay per use tests. Sometimes they have no research budgets, or if they do, they’re not large enough for that kind of thing. Our real goal is to create something useful for researchers.”
Through the Matrix Reasoning Challenge, Chabris and Engel also hope to better understand how crowdsourcing can be used to problem-solve in social and cognitive sciences.
Social scientists already widely use crowdsourcing sites like Amazon’s Mechanical Turk to recruit participants for their studies, but the matrix project is different in that it seeks to tap into the public’s expertise to help solve scientific problems. Scientists in computer science and bioinformatics have been able to harness this expertise to yield some incredible results. Using TopCoder.com, NASA was able to find a more efficient way to deploy solar panels on the International Space Station. Harvard Medical School was able to develop better software for analyzing immune-system genes. With The Matrix Reasoning Challenge, Chabris and Engel are beginning to explore crowdsourcing’s potential in the social sciences.
The beauty of crowdsourcing, Chabris says, is being able to tap into the knowledge of many unlikely experts. Anecdotally, Chabris says many of those who have already submitted puzzles don’t have Ph.D.s or backgrounds in psychology, but are really good at making puzzles.
“Psychologists are not really the right people to design puzzles. The people who are interested in puzzles are the right people to design puzzles,” Chabris said. “Psychologists are the right people to figure out what those puzzles are measuring.”
As of press time, they’ve received 120 submissions from over 40 individuals. Chabris and Engel are also committed to publishing a paper about the campaign. Each person who submitted a valid matrix puzzle that is used in their final test will be listed as an author on the paper.
They are offering $2,000 in total prize money awarded to the members of the crowd who submit the best and second best puzzles, the most and second most valid puzzles, and six puzzles at random. The contest is open until January 31, 2014. Submit your puzzle here: New Matrix Reasoning Challenge
UPDATE (3/15/14): The contest closed as planned on January 31, 2014. They received over 600 submissions that they are now evaluating.
