On Saturday, June 11, thousands of test-takers took the June 2022 LSAT administration. And for a while, at least, everything went according to protocol.
These test-takers logged in to ProctorU, connected with a proctor, and took three sections of the test we've all come to know and, if not love, then tolerate.
But in the fourth section, something very unusual happened. Some test-takers received an unexpected message on the load screen before beginning the final section of the test. These test-takers were told, "This section will be unscored."
These test-takers were explicitly told that the last section was an "experimental" section and that it would not count toward their final score. Implicitly, some of these test-takers received a message that they could take a load off, prop their feet up on their desk, and not worry about the final part of the LSAT.
Being told which section is experimental is strange enough, but what appeared in that experimental section was odder still. Except for a brief respite during the LSAT-Flex era between May 2020 to June 2021, every LSAT test has one unscored experimental section. However, these experimental sections are always another Logical Reasoning, Reading Comprehension, or Logic Games section. The experimental sections traditionally look like any other section, so test-takers can't discern which section is unscored while they take this test.
However, the test-takers who received the announced experimental sections we not treated to another Logical Reasoning, Reading Comprehension, or Logic Games section. Instead, the questions were allegedly new variations of Logic Games questions. Interspersed among these quasi-LG questions were survey questions, which asked about test-takers use of scratch paper on the Logic Games variations. Even without the announcement on the load screen, the test-takers would have likely deduced that something was amiss with their final section.
What Was Up with that Survey-Experimental Section?
So, why did the Law School Admissions Council (LSAC), the body responsible for constructing and administering the LSAT, include this survey-experimental section? And why would they announce to test-takers that their fourth section would be unscored right before those test-takers took the fourth section? Why not make a blanket announcement to all test-takers ahead of the test, or not announce this at all?
LSAC felt such an announcement was appropriate because most test-takers would quickly deduce the survey-experimental section was irregular and might believe that something was wrong with their exam. Presumably, the knowledge that LSAC intentionally inserted this survey experimental into this exam would make these test-takers less likely to suspect that something was wrong with their exam and that they may not receive a valid LSAT score. Rest assured, every test-taker who received the survey experimental—as well as the test-takers who did not receive that survey experimental—will receive a valid LSAT score if they completed the exam and submitted their Writing section sample without incident.
LSAC also felt confident that this survey experimental would not give an unfair advantage to test-takers given that section. Apparently, LSAC has years of data that suggest that the placement of the experimental section doesn't affect LSAT performance. Of course, LSAC doesn't have much data on whether telling test-takers they're about to take an unscored section affects LSAT performance. Nonetheless, LSAC felt this survey-experimental section would not give any test-takers an unfair advantage.
And why didn't LSAT just tell everyone that some test-takers would get this experimental section before the test? We're guessing they didn't want to induce hysteria among June test-takers by announcing this irregularity. Not that this ended up mattering . . .
How Did Test-Takers React?
Of course, those who received the survey-experimental section aren't complaining about the shorter exam. However, many test-takers who received the traditional, unannounced experimental section took to the internet and LSAC phone lines to complain—a reaction that is certainly understandable.
As you can imagine, when a bunch of test-takers spend months gritting through an oft-difficult study process, proceed to finish one of the most stressful tests of their young lives, and then learn about an irregularity like this … said test-takers will have some words. Certainly, something feels intuitively wrong that some test-takers were given this survey-experimental section while most were not. It seems unfair in a nebulous, hard-to-define way.
But do these criticism have merit? I don't want to link to frustrated test-takers' objections, but I'll do my best to paraphrase their positions. The most common complaint had to do with testing fatigue. According to these test-takers, those who received the survey-experimental section were less tired during their last scored section. Suppose that Test-Taker A got three scored sections and then the survey-experimental as their fourth section, while Test-Taker B received the normal experimental section as one of the first three sections. In that case, Test-Taker A took their last scored section after being fatigued by only two sections, while Test-Taker B took their last scored section after being fatigued by three sections.
This position is reasonable, but this line of reasoning would also apply to any test-takers who got their experimental section as their fourth section—whether it was the announced survey experimental or the usual unannounced experimental. Along the same lines, LSAC could have anticipated and allayed this criticism if they made everyone's experimental section the fourth section. (Based on test-taker reports, this seems unlikely.)
Frustrated test-takers also bemoaned the information imbalance. LSAC told some, but not all, test-takers which section was their experimental. Those who heard that the final section was experimental had more information about their test than other test-takers. Of course, such test-takers couldn't retroactively go back to the first three sections and improve their scores. But this information might make it easier to decide whether to cancel their score or purchase the score preview option. This worry is probably only compounded by the fact that LSAC has recently started to crack down on post-test speculation about which section is experimental.
While these concerns have merit, the through-line is that some test-takers received an unfair advantage. In my opinion, these understandably frustrated test-takers haven't convincingly proved that they were disadvantaged—that their version of the test made it harder to earn a given score. My suspicion is that their complaints may be partially motivated by a common misunderstanding about how the LSAT is scored.
Many refer to the LSAT as a "curved" test—bringing to mind the curved tests in college in which the student with the best score gets the best grade, the student with the worst score gets the worst grades, and all other students are plotted on a bell curve based on their scores. These "curved" tests are scary—you could, in theory, do well enough to earn an A on a non-curved test but still get pushed down the bell curve to a lower mark by your over-achieving peers. These tests can also make school seem even more competitive, as you are pitted against your fellow students to earn a limited number of top grades.
But the LSAT isn't "curved," at least not like that. (Pedantically, we'll note that it is technically "scaled.”) When you earn a 90th-percentile score on your LSAT, that doesn't mean you got more questions correct than 90% of the people who took the same LSAT administration as you. That 90th-percentile score means you earned a scaled score of 165, a score that 90% of test-takers historically can't attain.
To get that 165 on your test, you had to answer a certain number of questions correctly. And the test-writers figure out how many questions you had to answer correctly to earn that 165 before your test. (In a bit of irony, they do that by using data from past exams' experimental sections.) Your fellow test-takers can't really impact whether you'll earn that 90th-percentile score.
That is to say, the LSAT is not a zero-sum game. Theoretically, everyone could earn a 180 on a given LSAT. So, those who had the survey-experimental didn't disadvantage any test-takers on the June 2022 LSAT. Perhaps those test-takers were given a slight advantage, making a given score slightly easier to attain. But no more than any other test-taker who had the cosmic fortune to get an experimental section as their final section. And no more than those who took the LSAT-Flex between May 2020 and June 2021 and didn't have to worry about an experimental section at all. It might be good to keep this in mind if LSAC continues to use survey-experimental sections in the future. And speaking of the future . . .
What this Experimental Section Means for the Future of the LSAT
The remaining question is whether this survey portends major changes to the LSAT. Specifically, the elephant in the room is a 2019 legal settlement LSAC reached with law-school aspirants who were blind and visually impaired. These law-school aspirants argued that LSAC's available accommodations in the Logic Games section don't adequately address the needs of test-takers with visual impairment or blindness.
As you may know, the Logic Games section can be much more manageable with visual diagrams that you can make on your scratch paper. For obvious reasons, test-takers who are blind or visually impaired cannot make use of these visual aids. These law-school aspirants requested that LSAC remove the Logic Games section altogether.
Following the settlement, the attorneys representing the law-school aspirants announced that LSAC promised to remove the Logic Games section as a requirement for test-takers within five years. As this 2024 date nears, many wonder if this survey-experimental section is the first public step LSAC is making to replace the Logic Games section.
After all, the survey questions LSAC asked in this announced-experimental section were about test-takers' use of scratch paper on the new variations of Logic Games questions. Many are speculating whether LSAC is beginning to construct a new Logic Games section that doesn't require complex diagramming.
For what it's worth, we should mention that LSAC's announcement following the 2019 settlement was that they would "research alternative ways to assess analytical reasoning." Technically, LSAC never promised to remove the Logic Games section entirely. Perhaps, this survey-experimental section will be the only "research" into alternative ways to assess analytical reasoning. However, that seems a bit unlikely.
We would not be surprised if future tests continue to offer survey-experimental sections for some test-takers as LSAC continues gathering data about changes to Logic Games. However, if LSAC chooses to make significant changes based on this data, they'll certainly give us all more advance warning than they did for June 2022 test-takers.