“Biggest Bar Exam Disaster Ever?” was More a Fender Bender

Last week’s headline in Above The Law turned out to have somewhat exaggerated the lasting effects of an as-of-yet not fully explained technical glitch in software used in several states to adminster the bar exam. The software, a product called SoftTest, created and marketed by ExamSoft, is designed to let students type exam answers on their computers, while preventing access to any other programs on the computer or in the cloud. It also encrypts the resulting file, provides continuous time stamps, and prevents the student from revisiting completed answers. Before getting into the specifics of what happened on this occasion, it is in order to provide some background and context.

The bar examination is a high-stakes test, possibly the most high-stakes test in the country, that virtually any law student who wants to practice law must take and pass before being admitted to the bar. It is the culmination of a three-year professional school career that itself is filled with high-stakes end-of-semester tests, and it is expensive to both prepare for and take. Bar review courses, which are more targeted to test performance than are law school classes, can cost $1,500 – $3,000, and administration fees for the tests and the background checks required for bar admission can add from several hundred to more than a $1,000 to the cost. There is also the cost of transportation to and lodging in the city where the exam is administered. It should be obvious, then, that a law student about to take the exam is likely to be under a lot of stress and anything that goes wrong with the test administration will further exacerbate that stress.

Law schools began letting students take semester exams on their computers in the early 1990’s. That move was very controversial, as despite the existence of an honor code at most law schools, faculty and administration were concerned about their perception of an increased likelihood of cheating by students’ accessing other materials stored on the computer. Companies such as ExamSoft began marketing so-called “secure exam programs” to law schools, and this relieved much of this concern. Over the years, there were four companies in this market: ExamSoft, Extegrity, Software Secure, and Electronic Bluebook. The name of that last product refers not to the common name of A Uniform System of Citation but rather the small format exam booklets into which students typically wrote their exam answers. As the years passed, more and more law schools allowed students to take exams using one of these packages. Technological problems sometimes occurred, resulting in very upset students and administrators, but very rarely was an exam answer lost. Anyone who regularly works with computers knows that any word processor will sometimes crash, often resulting in the total loss of the work to that point. The exam software packages all made automatic backups every minute, so if they were performing properly, the student was never at risk for losing more than a minute’s worth of typing.

Once the exam software companies had signed up most law schools, they turned their attention to the bar exams. The bar exam in each state is administered independently from that in other states; however, most states require the student to pass portions of the exam that are offered on a national basis: the Multistate Bar Examination, the Multistate Examination, the Multistate Performance Test, and the Multistate Professional Responsibility Examination (MPRE). For a detailed state-by-state breakdown of this, see Chart 8 in the ABA’s 2014 Guide to Bar Admission Requirements. Other than the MPRE these exams are offered twice per year, at the end of February and again at the end of July. They are administered on the same dates on all the states that use them. The bar examiners’ boards in most states were also slow to allow students to use computers to take their exams, but the marketing by the secure examination software vendors accelerated the adoption of computer-based exams.

In 2007, during the summer administration of the New York bar exam, Software Secure’s SecureExam malfunctioned. In an interview with John McAlary, executive director of the New York State Board of Law Examiners, April Dembosky reported  in a July 29, 2008 post to the City Room blog of the New York Times that, “Last summer, panic spread through various testing sites when the exam software – which locks down all programs and files except the exam – malfunctioned. Hundreds of laptop users who navigated back to a previously completed essay found a blank screen. In the months following the exam, Mr. McAlary said, the board salvaged all but 47 essays. Some of those candidates passed or failed regardless of their score on the lost essay, leaving only 15 that were given an estimated score. Nine of them passed the bar, six didn’t.” Following that event New York switched to using ExamSoft.

Against that background we can look at last week’s malfunction in many of the states that use the ExamSoft product. Apparently the program could not upload the completed exam to ExamSoft’s servers at the conclusion of the exam. This problem continued into the evening and overnight hours following the Tuesday exam administration. Whether or not the students were aware of it, their exam files were complete and safely stored on their computer’s hard drive. But the bar examiners have very narrow deadlines for uploading the exams; e.g. in Ohio the deadline was 10:00 P.M. on Tuesday. There were delays in contacting students with information concerning relief from the deadlines, and it is not now clear what, if any, protocols were in place at bar administration sites for communicating with students after problems had been discovered. As a result, students were stressed, pissed off at Examsoft given they had no helpful information, and the complaints quickly spread on social media and sites such as Above the Law.

Several of the technical personnel who assist with the administration of bar exams by computer are law school technology staff who contract with the exam software companies to serve at the bar exams. Many of them have been posting messages concerning last week’s events on Teknoids, the mailing list of law school technology folks. You can view their posts on the list’s archives and you will find that all of them report that no exam files were lost last week, and ultimately all of them were successfully uploaded.

So I call this the equivalent of a fender bender: while it’s happening it’s scary and you don’t know how it will end. Your stress level skyrockets. You may lose hours out of your otherwise productive schedule dealing with the aftermath. But in the end, you’re uninjured and you get your car repaired. Moving forward, ExamSoft has a duty to post a full post-mortem report explaining the cause of the problems and the steps the company is taking to prevent a reoccurrence. But no company will ever be able to guarantee 100% error-free computing. The boards of bar examiners also have a responsibility to properly respond to this event. First and foremost, they need to adopt and publicize protocols that set forth channels of official communication concerning deadline extensions and other remedies that will be offered during any future similar circumstances. The first thing they should consider is relaxing the deadlines in advance of a test. The software encrypts the exams and applies a timestamp to it. There is very little risk of anyone altering the exam file prior to its being uploaded. The bar examiners already set up the system that maximizes stress; they should seize this opportunity to dial it down a notch.

Leave a Reply

Your email address will not be published. Required fields are marked *