Auto Review best practices for writing prompts and sampling

Drafting Case Background and Tag Descriptions (Prompts)

• Be concise: lists of key points work better than paragraphs.
• Use narrow language. Terms like “explicitly showing” or “specifically contains” work better than broad terms like “related to” or “may indicate.”
• Do not include explanations of law.
• Use headings where applicable. For example, in your Case Background, headings like “definitions:” and “background:” are helpful.
• When defining a key entity, include any aliases or nicknames commonly used for that entity.
• Include language specifying the scope of application. For example, Case background prompts should state that the definitions and background information apply generally to all tags in the review.
• Focus on document content rather than degree. For example, instruct Auto Review to tag a document as Confidential if it is a board presentation rather than if it includes sensitive nonpublic information.
• Input tags in the order they should apply. For example, if a document is only responsive if one or more of the issue tags applies, run the issue tags first, then instruct Auto Review to apply the responsive tag if an issue applies.
• Log tag descriptions used and results generated for each round of prompt iteration in a spreadsheet for tracking and validation.
• Use a field for each tag to log prompt iteration notes, like trends in Auto Review’s suggestions for that tag and required prompt updates. 
• Break prompts into parts. For example, instead of looking for “communication between plaintiff and defendant about a widget dated 1/1/2023 or later,” try:
  • Apply this tag only to documents that explicitly show all of the below parts:
    • Part 1: A communication:
    • Part 2: between plaintiff and defendant
    • Part 3: about a widget
    • Part 4: dated 1/1/2023 or later
• Instruct Auto Review to provide citations for its suggestions to assist in prompt iteration. For example, “Provide the plaintiff and defendant in the document, the specific widget, and the source of the date provided in Part 4 above.”
• When drafting a list in a prompt, use a single dash (“-”) to indicate the top level item of the list. Use a double dash (“--”) to indicate a sub-list item.
• Use a defined list to refer to definitions that include numerous items. Example:
  • “Relevant Widgets list” includes:
    • Widget01
    • Gadget02
    • Gizmo03
    • Contraption04

Sampling

• For low-prevalence populations or issues with low tag rates, enable Cecilia Q&A on the database or TAR AI on the tags and run targeted searches to identify additional documents warranting the tags. Then use the results of the searches and high-scoring documents to enrich the alignment samples. 
• Stratify alignment samples by document type to best reflect the underlying data set. For example, if your Auto Review population is 40% emails, your alignment sample should also be 40% emails.
• For very low prevalence populations, choose a reasonably large control set and target a higher recall margin in prompt iteration to account for the larger error margin in the control set.

Example Case Background and Tag Descriptions

Below are examples of prompts loosely based on the Enron matter. Please note that these are fictitious and not true to the case.

Case background

Definitions:

• Kenneth Lay was the Founder, CEO, and Chairman of Enron. Lay was accused of creating a culture that fostered fraud. 
• Jeffrey Skilling was Enron’s CEO before Lay returned. Skilling was instrumental in creating Enron's aggressive trading culture and pushing the accounting manipulations.
• Andrew Fastow was Enron’s CFO. Fastow was the mastermind behind many of the SPEs and other complex accounting schemes.
• SPV or SPE means special purpose vehicles or special purpose entities, entities used by Enron to hide debt or toxic assets
• MTM means mark-to-market accounting, which allows a company to adjust the value of its balance sheets from their historical value to their fair market value (FMV). Mark-to-market accounting allows for income to be calculated as an estimate of the present value of net future cash flow. Enron used mark-to-market accounting to hide their debt and manipulate their financial outlook.

Parties:

• Defendants refers to Enron Corp., Arthur Anderson LLP, Kenneth Lay, Jeffrey Skilling, and Andrew Fastow.
• Plaintiffs refers to Mark Newby and Amalgamated Bank.

Email domains: 

• Any Enron party has an email domain enron.com. 
• Arthur Anderson has the email domain arthuranderson.com.

Background:

Enron was formed in 1985 through the merger of two natural gas pipeline companies. Under the leadership of CEOs Kenneth Lay and, later, Jeffrey Skilling, Enron transformed into an energy trading and commodities company. It expanded aggressively into various markets, including electricity, natural gas, broadband services, and even weather derivatives. Enron Online, its online trading platform, was executing trades worth billions daily. Enron's executives, primarily led by Chief Financial Officer (CFO) Andrew Fastow, used highly complex and deceptive accounting practices to manipulate the company's financial statements, including mark-to-market accounting (MTM), special-purpose vehicles (SPVs), and “round-tripping.”

Tag descriptions

Mark-to-Market Accounting

This tag includes any documents demonstrating Enron's use of MTM accounting or any documents discussing MTM accounting. This includes but is not limited to discussions regarding whether or not MTM accounting should or should not be used, whether MTM accounting has been used, and explanations or illustrations of the ways in which Enron has used or will be using MTM accounting.

One example of Enron's use of MTM accounting is found in the partnership between Enron and Blockbuster, a 20-year deal to sell movies on demand via a broadband network. The use of MTM accounting allowed Enron to include all twenty years' worth of the deal's forward projections in its financial statements for a single year.

Special Purpose Vehicles

This tag applies to any documents discussing or exhibiting the use of SPVs by Enron. Some examples of SPVs include raptor, talon, timberwolf, bobcat, and porcupine. Some of the deals involving SPVs include the Thomas and Condor tax shelters, the Apache deal, the Steele deal, and the Cochise deal. The special purpose vehicles were not controlled by an independent party; instead, they were substantially a part of Enron, and should have been included in Enron's balance sheets. This tag applies to any information related to whether the special purpose vehicles should or could be either included or not included in Enron's balance sheets. Additionally, this tag applies to any guarantees made to Andrew Fastow or other equity holders related to the SPVs.