By organizing documents into review “stages,” you can control user access to each stage, customize the review decisions, and monitor the progress of each stage. This article will cover best practices on how to review the opposition’s production(s) for issues.
Ingest, create tags, and organize documents
First, obtain the opposition’s production and send the data to DISCO for processing and ingest. During ingest, DISCO strategically deduplicates your documents, extracts OCR data, and identifies family and conversation relationships among all applicable documents.
Next, create tags for various issues that you expect to find in the documents. In the main menu, select "Tags" under "Setup." From this page, you can add and delete tags, edit the shortcuts for tags, turn on and off tag predictions, and add and delete tag groups.
After your documents are ready for review, create your review stages. To add a stage, navigate to the top menu WORKFLOW > Manage Review Stages. From here, create a new stage for your review. Common stages include:
- Issue Review - all or some documents that need to be reviewed for issues. If you receive documents as a rolling production, where data comes to you in stages rather than in one piece, set up your stage groupings by ingest session and group by Bates number.
- Escalation Review - funnel documents from the issue review into an escalated review stage for attorneys with more familiarity with the facts and issues of the case.
a. Populate Stages with Documents
For each stage, begin by populating it with sources. The default source is “all documents”. Should you choose to focus your stage on a subset of all documents, you can populate the stage with search results as sources. You may have up to 20 discrete sources per stage each containing a distinct search string. For example, you can use a search string that limits the documents in a stage by a date range, to a particular set or custodians, or to a set of search terms. To make sure that all family members are included in a batch, when setting up the stage, turn the toggle ON next to "include family members of search results.” If you would like to include conversations and group by them in your batch, turn the toggle ON next to "Include conversations".
b. Organize Documents
Organize your documents in three ways: by group, order and batch. You can restrict access level permission control for groups. You may group by custodian, document type, ingest session, stage source, or have no grouping. Order adjusts the sort order within each group (if you have used groups). If you chose to NOT keep families or conversations together in your batch, you may select: family, document date, Bates number, or path. If you chose to keep families together, the default order is 'Family Date'. If you chose to keep conversations together, the default order is "Conversation Date". (Note: when keeping conversations together, documents not part of a conversation will be ordered by their family date if in a family or by their document date if not.)
Finally, pick your maximum batch size. Batches represent the maximum number of documents that may be checked out in a set of documents. Users may “check out” multiple batches--though if the documents remaining in a group are less than the maximum batch size, the batch will only contain the available number of documents. Five hundred documents is a common batch size. Batches are checked out by reviewers but may be re-assigned to a different reviewer by an administrator.
Once you’ve selected your ordering, you can then “Rank” your groups by dragging and dropping the rows. Ranking places the visible instance of deduplicated documents in the highest ranked group. Ranking is important for getting the most context out of your documents you are reviewing. For example, if you group your documents by Custodian (as displayed in the image above), you will want the most 'important' Custodian listed on top. You will want to rank them this way so that if you have documents across multiple custodians your documents will get grouped into your review stages in such a way that you get the most context around those documents.
Create Decision Pane
Next, create a decision pane or "Review Decisions". Review decisions are the decisions that you want the reviewers to consider when reviewing the documents within a stage.
Construct your decision pane by clicking the buttons shown above and selecting your field type. Let’s say we’re building an issue review stage. Click the “select multiple tags” component. Remove privilege tags as options and save. Once you click on a component, it will appear in the decision pane editor on the right. You can include a label or instructions, such as “Issue Coding” by typing in the box. Then you can pick which tags you’d like the reviewer to choose from and the radio buttons will be added to the decision pane. As you build the pane editor, it previews the completed review pane exactly as the reviewers will see it when they review the stage’s documents.
Continue to add components to your decision pane until you are satisfied with the result. Here, we’ll add a document note and if you want to create an escalation review stage, the optional checkbox “needs further review”.
Add & Assign Reviewers
Once you've set up your stages and created the decision pane, create users and assign them to each stage. Navigate to the main menu and select "Review team."
From here, add a new user. You'll have three roles to choose from (custom roles are available as well):
- Admin: The admin role may complete all actions in DISCO.
- Reviewer: The reviewer may complete searches, complete batch actions such as mass tag and exhibit sets, and produce documents from DISCO as well as review documents inside and outside of Review Stages. Reviewers, however, do not have access to setup stages nor to manage stages.
- Restricted Reviewer: The restricted reviewer does not have access to the usual DISCO features of search, batch actions, and productions. They only have access to the stages for which they have been assigned. In the document viewer, restricted reviewers will have access to view related documents (such as families, conversations, and similar documents). However, if a related document is not in one of their assigned batches, the restricted reviewer will not be able to perform any action on the document; they will only be able to view these related documents for purposes of context in their review.
Each time you add a new (restricted) reviewer to a stage, they will receive a welcome email instructing them how to login to the database and start reviewing. Once restricted reviewers are added to a stage, they can then begin checking out batches and reviewing documents.
Checking Out a Batch
A Reviewer will navigate to the DISCO main menu WORKFLOW > Check Out Batches to check out a batch. Each batch is a collection of next available documents or from a group you chose to organize by (default is set to "Next available". Lastly, you will click the blue "Check out" button.
Reviewers can simply start by opening the first document in their batch, making the decisions pre-designed by the review manager, and click "Mark Reviewed & Next". A green box will appear on the right of each document showing it has been reviewed. There is also a progress bar showing how many documents have successfully been reviewed. Finally, once you are done reviewing your batch, you will check your batch back in by selecting the "Check-in" button.
*IMPORTANT NOTE(Checking batches out): When you check out a batch, DISCO uses "just in time batching" which means all of the documents that may be sitting in a stage are not put into batches until the moment a reviewer requests them. This means any change in strategy in your review does not revolve around lots of deconstruction of batches that have not yet been reviewed.
*IMPORTANT NOTE(Checking batches back in): For databases created BEFORE 02/07/17, a document enters "stage complete" after being marked 'Reviewed', regardless if the document was checked back in or still checked out within its batch. For databases created ON or AFTER 0207/2017, a document will enter "stage complete" ONLY IF the document has been marked 'Reviewed' and is currently in a batch that is checked in.
On the front page of the Review Stages tab, Admins can track the progress of the review stage, particular reviewers, groups or batch progress. Within the Metrics » Findings screen, you can view the overall distribution of tags of all the docs in a given stage. Further, you can click on any tag to view the documents reviewed in the stage with that particular tag. Furthermore, you can filter the chart to only show tags which were available to the reviewers for this particular stage.
* = All reviewer progress, regardless of their role, is tracked when using Review Stages. However, regular reviewers and admin may access documents in DISCO through the search. This activity outside of Review Stages is not tracked within the Stages progress screen.
To analyze the accuracy of your reviewers, QC the different stages as follows:
- Setup the tags “QC approved” or “QC defect” under the top menu Manage > Tags.
- Run a “sample()” search in DISCO to review a subset of the documents that have passed through the first stage. For example: sample(20%,"stagecomplete(FPR)"). This would give you a randomized sample of 20% of the documents that have completed First Pass Review.
- Drill in deeper to see documents tagged by a specific reviewer using the search "sample(10, tag(by email@example.com))".
- (Optional) Mass tag your search results with a tag such as “QC Sample 1” to save and access your QC batch at a later time.
- Review the sample results and tag as necessary to identify any problem documents.
- Determine extensiveness of QC issues and plan for resolution.
- Review and tag with correct tags. If there is an extensive problem, such as a general misunderstanding of a tagging instruction, the review manager should craft a new stage with a document source search that captures the affected documents to be reviewed again or reviewed by higher level reviewer.
Use the Documents
Once the review is complete, members of the case team can run a search on any desired issue tag, review them in DISCO, and use them outside of DISCO by adding some or all of the appropriately tagged documents to Exhibit Sets (read more >). Alternatively, you can Batch Print documents directly from DISCO (read more >).