What is Cecilia Auto Review?

DISCO's Cecilia Auto Review (or simply Auto Review) uses a large language model (or LLM) to suggest whether a document matches a written description provided by a person. Auto Review also uses generative AI to provide written rationale for its suggestion.

Auto Review makes its suggestion about the document based on the text of the document. It does not take into account pictures, videos, or metadata from the document. If a document has no text, then the document is not assessed by Auto Review. Auto Review does not take into account information about other related documents, such as a document's family members--Auto Review assesses a document on its four corners.

While Cecilia Auto Review can be used for any tag, it is not currently optimized for privilege review. For now, we recommend that its best use is for responsiveness and issue tags.

 

 

Terminology

There are a few new terms to know regarding Auto Review. Here's a quick rundown:

• Suggestion - This is a simple Yes or No assessment provided by Auto Review for each document for each tag included in Auto Review. If a tag is positively suggested for a document, that means Auto Review determined that the human-provided tag description is fulfilled by the document.
• Suggestion Reason - This is the written rationale provided by Auto Review for each suggestion. A reason is provided for both Yes and No suggestions. Failed and skipped documents do not receive suggestion reasons.
• Tag Description - This is the human-written description in which the person provides the details of what they are looking for in a document.
• Failed Document - If a document was unable to be reviewed by Auto Review, it is noted as a failed document.
• Skipped Document - This is a special case of a failed document. Skipped documents are tracked separately from failed documents. This designation is specifically for documents with no text, and thus with nothing for Auto Review to analyze. The search syntax textLength(0) locates documents in DISCO with no text.

 

 

Enabling Cecilia Auto Review

Auto Review must be added to your DISCO database before it can first be used. Please contact your CSM, DISCO Project Manager, DISCO Account Executive, or the DISCO Desk (discodesk@csdisco.com) to inquire about activating this system on a database and about obtaining pricing and workflow information.

 

 

 

How do I run a new review job in Cecilia Auto Review?

After Auto Review has been added to a database, a new icon will appear in the right-side options bar, as seen below. Clicking that bar will open the Cecilia sidebar. Users with the database role permission of Admin are able to run Auto Review jobs, as are users in a Custom role with the correct permissions. In order to run an review job in Auto Review, you will need to have a sufficient number of documents activated (paid for), which is achieved by working with your DISCO representative.

Your first step is to set up your population of documents to be reviewed. Auto review runs from being pointed to a set of documents within a DISCO folder. Therefore you will simply need to (1) add the exact population of documents to a folder, and then (2) select that folder using the folder browser located to the left of the search bar, seen below.

 

Select your folder using the folder browser, with the Include documents in subfolders toggle turned off:

And verify that you now have the desired set of documents:

 

It is important to note that the set of documents to be reviewed is entirely dependent on your selected folder in the folder browser. Currently, other restrictions added to a search, such as terms entered in the search bar or search pills added by clicking checkboxes in search filters, are not intended to change the set of documents that will be reviewed by Auto Review, even though your search total changes. If you wish to review a smaller selection of the documents from your selected folder, then you should (1) create a new folder containing only the desired set for review, and (2) point Auto Review to that new folder via the folder browser. You should ONLY use a folder to direct Auto Review to the set of documents for its review.

Once you have the desired set of documents, you need to choose your tags and enter your tag descriptions. If Auto Review has not yet been run in this database, then no tags will be selected yet. If Auto Review has already been run at least once, then the tags from the most recent job will be pre-selected, and the tag descriptions will be auto-filled. Of course, you may need to change which tags you want to have reviewed, and/or revise the tag descriptions being used.

You can select up to 10 tags from the database to be included in a single review job. Select them via the Tags to review dropdown menu, seen below. If the tags you need have not yet been created in the database, then they will need to be created (Menu/Tags) first.

Each tag you select in this menu will create a box for entering its tag description. A single tag description is allowed to include up to 2,000 characters, and there is a shared maximum of 10,000 characters across all tag descriptions in a single review job. In practice so far, we've observed that most people do not come anywhere close to these limits.

 

Once you have selected your desired tags and entered their tag descriptions (and double-checked them!), all that remains to run your review job is to click the Review X documents button at the top of the Cecilia sidebar, as seen above. If the button is grayed out, then you should hover your cursor over the button to receive a message with details about what still needs to be done (perhaps your folder contains more documents than have been paid for, or a tag description has not been entered yet, etc.). If the button is teal (i.e., active), then it is clickable; clicking the button will take you to a final confirmation window before actually committing. Running an Auto Review job incurs cost, so it's important to confirm that everything is correct before proceeding.

After the Auto Review job has been started, it can be canceled while it is in progress. However, any documents that have already been processed during this job will count against your billed total. Press the Stop this review button to end the in-flight job.

 

Once the Pending count reaches 0, all documents have been reviewed. However, it will take a couple of additional minutes for the final steps to complete--DISCO simply needs a little bit of time to finish adding the rest of the information to the database. The purple In Progress indicator icon will then update to a green Complete indicator icon.

 

Why is my Review X documents button grayed out?

This means that at least one necessary condition for running a new Auto Review job has not been met. Hover your cursor over the button to receive more information. To run an Auto Review job, the following conditions are all necessary:

1. In the folder browser, select the folder containing your intended-for-Auto-Review population of documents;
2. Select at least 1 tag to review (maximum of 10 tags);
3. Enter a tag description for every tag selected (minimum 10 characters per tag description, maximum 2,000 characters per tag description, and maximum of 10,000 characters shared);
4. Have a sufficient number of documents paid for Auto Review for this database (part of your SoW with DISCO);
5. Have sufficient user role permission in the database to run an Auto Review job.

 

How do I access Cecilia Auto Review's results?

Auto Review does not apply tags to documents. Instead, it provides suggestions and suggestion reasons. The suggestions and their reasons are associated to the tags that were used during the Auto Review job.

Importantly, there is a distinction to understand regarding suggestions, and this distinction only becomes apparent once a document has been reviewed for a tag during more than one job of Auto Review: the difference between the most recent suggestion for a tag for a document versus any historic suggestions for a tag for a document.

DISCO stores ALL tag suggestions (which can be either Yes or No) for a document for a tag as long as that tag and document exist in the database. As a corollary, it is important to not delete a tag from the database if you are using that tag for Auto Review (though not to be confused with merely unapplying a tag from a document, which is fine). If a document was reviewed for a tag during multiple different Auto Review jobs, the suggested Yes or No decision from each of those jobs can be searched for; this process will be described further below. This can be very important because you'll likely be revising tag descriptions to improve them during the early phase of your workflow, and thus it is useful to be able to access what those historic suggestions were.

Auto Review also generates reasons for each of its suggestions. Unlike the suggestions, the reasons are only automatically maintained for the most recent suggestion for each tag for each document. For example, suppose documents 1 and 2 are reviewed for tags A and B, and then in a second job, documents 2 and 3 and reviewed for tags B and C. The following chart details if any item will be automatically stored:

 

 

 

Seeing suggestions and suggestion reasons in the document viewer

Auto Review's most recent suggestions, and the reasons for those suggestions, can be easily seen while reviewing a document. DISCO's document viewer layout is slightly different depending on if you are viewing a document from within a review stage batch versus simply accessing a document from the main search and review screen, but both experiences provide full access to the needed information.

 

Shown below is an example from the document viewer in a review stage batch. Within the coding panel on the left, take note of the info icon on the right side of the tag. The info icon appears for every tag that has been included in an Auto Review for this document.

In the example, the Hot tag was not Auto Reviewed for this document. Both the LJM and Raptor tags were Auto Reviewed for this document, and were both positively suggested as tags--their icon has a green highlight to indicate they were positively suggested during their most recent Auto Review. The California Energy Market tag was Auto Reviewed for this document, but it was not positively suggested as a tag--its icon has a gray highlight to indicate that it was not positively suggested during its most recent Auto Review.

Finally, hovering your cursor over the info icon will display the suggestion reason.

 

Outside of review stage batches, the viewer's panel is slightly different. However, information is provided in a very similar way, as shown below. The info icons are used in the same way, with the green/gray distinction and the ability to hover-over for the suggestion reason. However, you'll notice that only the human-applied tags are shown in the top tags panel, and only the human-unapplied tags with positive suggestions are shown in the lower Cecilia Suggestions panel.

The human-unapplied tags with negative suggestions can still be accessed here, but DISCO is trying to avoid information overload. To access these additional tags, simply click the top tags panel to open the dropdown menu containing a list of all unapplied tags, as shown below.

 

 

Seeing suggestions and suggestion reasons in the document list

Auto Review's most recent suggestions of Yes/No for each tag can be added to custom views in your document list. The options for the two suggestion columns, Suggested as likely and Suggested as unlikely, can be found in the Work Product/Tags section of the custom view builder, seen below.

Next, let's take a brief look at how the information from Auto Review is displayed in these columns.

In the below example, there are four documents shown, each of which has gone through Auto Review for three tags.

In the first document, the Word doc with ID 174257, two tags were positively suggested, and one tag was not suggested. One of the likely tags, Raptor, was also applied to the document by a human reviewer--this can be seen in the Applied Tags column, and also by the tag being colored-in when it appears in the suggestion columns. The color of the shading corresponds to the color of the tag group in DISCO, which is teal in this example.

In the second document, the Word document with ID 165538, all three tags were suggested as not being applicable; you can also observe that one tag (LJM) has been applied by a human reviewer (the tag is colored-in). These first two documents are also good examples of disagreements between the tagging applied by a human reviewer and the suggestions provided by Auto Review--they disagreed regarding the LJM tag. In the first document, the human reviewer did not apply LJM, but Cecilia has suggested it. In the second document, the reverse has happened--the human reviewer applied LJM, but Cecilia did not suggest it.

 

Additionally, the reasons provided by Auto Review for its tagging suggestions can be seen by hovering your cursor over the tag in the suggestion columns, as seen below.

Finally, Auto Review's most recent suggestions and suggestion reasons can be exported to an XLSX file via DISCO's document list export feature. They will be included in a column listing three items: (1) the tag name, (2) the suggestion (either yes or no), and (3) the suggestion reason; these are semicolon-delimited, with double semicolons separating each of these trios if multiple tags have been Auto Reviewed for a document.

 

 

Finding suggestions via searching

DISCO allows you to find the specific suggestions from Auto Review through its search engine. To that end, the Auto Review side panel's results are all searchable links! Clicking on any of the numbers in the Auto Review side panel will bring up the set of documents underlying that number.

For example, opening the Overview tab in the Auto Review side bar will bring up the list of all of the tags Auto Reviewed in that job. Each tag shows the tally of documents suggested during that job. Clicking that number will add a search to the search bar with the exact syntax to find the set of documents. The syntax follows this format: aitagdecision() with 3 parameters: (1) the name of the tag, (2) Y (for positively suggested) or N (for not suggested), and (3) a unique identifier for the job.

Note that while the document list and viewer will always show the most recent suggestion for a tag for a document, the search index has ALL of the suggestions available. This can be critical if you've changed the tag definition for a tag and run it again in a new job, and you want to understand which documents changed from being not suggested to suggested for a tag, or vice-versa. And because you won't always have the perfect tag description the first time every time, it can be very useful to test a few iterations on a sample of documents.

Pro Tip: replacing the second parameter's Y with Y or N will find all documents which were Auto Reviewed for that tag during that job, instead of only those which received a positive suggestion during that job.

 

(Currently, Auto Review's result searches are not enabled in DISCO's Search Builder. Thank you for your patience--this one's on our list! We understand the utility of combining several of these tag suggestion searches into one search. For right now, if you're building out a larger search incorporating several Auto Review suggestion searches, we advise opening a second browser tab of your database to allow for quick copy-pasting.)

 

 

Comparing Auto Review's tag suggestions against tags applied by a human

Each Auto Review job that has been run in a database is accessible from its card in the Auto Review side bar, shown below. Each card's details include how many documents were submitted, start and end times, the person who ran the job, and the total run time. If some documents from an Auto Review job are later deleted from a database, this top level card will remain unchanged; the searches and statistics discussed later in this section, however, will change to reflect the loss of those documents.

 

Clicking into the card for a specific job reveals a bevy of information across three tabs. The Overview tab  contains two sections.

First, the Documents overview section contains search-clickable tallies for the documents that were successfully Auto Reviewed, for the documents that Failed, and for the documents that were Skipped (skipped documents have no text). Failed and skipped documents must be manually reviewed--Auto Review returns neither a Yes nor a No decision for those documents. We also advise that lengthy documents be re-reviewed manually--this advice will change as LLM technology continues to rapidly advance, but we currently consider it good practice to manually re-review documents larger than approximately 30,000 characters, textLength(>30000).

Second, the Documents by tag section in the Overview tab contains search-clickable tallies for the positively-suggested tag descriptions for the documents in that job. Importantly, these search links are for the tag suggestions from that specific job's tag description. It's possible (and highly likely) to run Auto Review for a tag in multiple jobs, but using a different tag description. That's because it's usually a good idea to test tag descriptions on a small sample of documents that you've already reviewed yourself, so that you can assess if the tag description is correctly capturing the attributes you're looking for, and then modify the tag description based on the discrepancies you observe.

The Tag guidelines tab provides a record of the specific tag descriptions that were used for that job.

 

The final tab to discuss, Compared to human review, has a large amount of information available. As the name implies, the purpose is to compare the suggestions provided by Auto Review to the tags that have been applied to documents by human reviewers. These metrics are ONLY useful for sets of documents that have also been reviewed by humans. Otherwise, what are Auto Review's suggestions being compared against?

 

The comparison page defaults to using All documents, and can be switched to use your current Search criteria, as shown below.

When using the All documents option, the comparison tool uses the full set of documents that were included in that specific Auto Review job (excluding the skips and fails). When using the Search criteria option, the comparison tool limits the set to the overlap between (1) the full set of documents that were included in that specific Auto Review job (excluding the skips and fails), and (2) the set of documents that are included in your current search in the search and review screen, including combinations of syntax in the search bar, a folder selected in the folder browser, and any search pills added from the search filters panel.

The Search criteria limiter is very powerful. In technology-based workflows, it's usually very important to review samples of documents. By using the Search criteria limiter, you can immediately receive the metrics for the overlap (i.e., intersection) between your Auto Review job and your desired sample.

 

Overall metrics

The Overall metrics section provides the metrics for all of the tags that were Auto Reviewed in this specific job. For values that are averages, these are specifically macro averages, meaning that the value from each tag is added, and then divided by the number of tags (as opposed to a micro average, which allows different weightings).

What does each of these metrics mean?

• Agreement rate - the percentage of Cecilia's decisions that agree with your team's decisions.
  • This is an average value from all of the tags reviewed in this job
• Comparison - the amount of documents that were reviewed by this Auto Review job and which are also present in the set of documents being compared
  • Note: this amount will always be 100% if the set of Documents to compare against is selected as All documents
• Prevalence - the documents for which Cecilia suggested at least one tag
  • This is NOT an average value
• Precision - when Cecilia positively suggested a tag, how often was it right?
  • "Being right" here means that for documents where Cecilia suggested the tag, the document also has the tag applied to it by a human
  • Note: these values will update (when you refresh your screen) as tags are newly applied or unapplied
  • This is an average value from all of the tags reviewed in this job
• Recall - when a document has a tag, how often did Cecilia suggest that it should have that tag?
  • Note: these values will update (when you refresh your screen) as tags are newly applied or unapplied
  • This is an average value from all of the tags reviewed in this job

Caution! Be careful when clicking the links for the comparison metrics. Clicking the link adds the logic to the search bar, which means that if you have the Search criteria limiter in use, your search will update, and thus your displayed metrics will also update.

 

Agreement rate by tag

The Agreement rate by tag section allows you to drill down into the metrics for each tag for that run. Five percentages are provided, along with an error matrix, which is the two-by-two grid comparison chart of Auto Review's suggestions versus the currently-applied tagging.

At the top level, next to the name of the tag, the accuracy metric is provided. In the example above, this is 94% for Raptor. This is calculated by tallying all of the agreements and dividing that by the total number of documents. In the example, accuracy is (5+11)/(5+1+0+11) = 16/17 = 94%.

Moving downward, prevalence is the documents for which Cecilia suggested the tag versus the total number of documents. This is calculated by tallying the documents for which Cecilia suggested the tag and dividing that by the total number of documents. In the example, prevalence is (5+1)/(5+1+0+11) = 6/17 = 35%.

Next, precision measures how often Cecilia was correct for the documents in which Cecilia positively suggested the tag. This is calculated by dividing the Yes-Yes (the true positives) agreement value (top left) by the sum of the top row. In this example, precision is 5/(5+1) = 5/6 = 83%.

Next, recall measures how often Cecilia suggested the tag within the set of documents that do have the tag. This is calculated by dividing the Yes-Yes (the true positives) agreement value (top left) by the sum of the left column. In this example, recall is 5/(5+0) = 5/5 = 100%.

Finally, the F-1 score is the harmonic mean of the precision and the recall. This is calculated by adding the reciprocals of the precision and the recall, dividing by 2, and then taking the reciprocal. Don't worry, we did the arithmetic. In this example, the F-1 score is 1/([1/(5/6) + 1/(5/5)]/2) = 1/([(6/5) + (5/5)]/2) = 1/([11/5]/2) = 1/(11/10) = 10/11 = 91%.

These values can be quite useful for understanding and defending the effectiveness of your review.

 

It's also impossible to overstate the utility of the two disagreement search links, displayed in watermelon red in the error matrix. Those links will take you directly to the documents where Auto Review's interpretation of the tag description did not match how the tag was applied (or not applied) by a human reviewer. Understanding those disagreements can be critical to (1) refining the tag descriptions, or also for (2) finding possible errors made during human review.

 

 

 

Permissions - restricting and granting access to Cecilia Auto Review's aspects

Sometimes it's important for access to some information or some actions in a database to be restricted, and Auto Review is certainly no exception. To that end, there are several options for granting/restricting access to (1) running Auto Review and (2) accessing Auto Review's results. Most of the permissions are built into DISCO's user roles permissions, and there's also added granularity when setting up review stages.

 

DISCO's user roles can be accessed by Admin-level users via Menu/Review Team.

There are two new permission rows present when Auto Review is activated in a database, Explore/Cecilia Auto Review and Document Viewer/View Tag Suggestions, shown below.

The Explore/Cecilia Auto Review permission has three levels: (1) unchecked, (2) view, and (3) manage. The unchecked level means that a user will not have access to the Auto Review side bar on DISCO's main search and review page. The view level means that a user will have full access to the Auto Review side bar, but will not be able to run new Auto Review jobs. This user will have full access to all of the job cards, metrics, and search links, but there's no ability to run a new job. Finally, the manage level means that a user has full access to Auto Review, including the ability to run a new job.

The Document Viewer/View Tag Suggestions permission has two levels: (1) unchecked and (2) checked. This option activate/deactivates the indicators for suggestions and suggestion reasons within the document viewer panels.

 

Within DISCO's three default roles of Admin, Reviewer, and Restricted Reviewer:

• Admins have manage permission
• Reviewers have view permission
• Restricted Reviewers have no permission (i.e., the box is not checked)
• All 3 default roles are checked for View Tag Suggestions

Finally, when setting up a review stage, the ability to view tag suggestions within that stage's batches can be changed via the Suggested tag details toggle in the Preferences section of the Edit review decisions page, shown below.