5 Things Every Legal Pro Should Know about Managed Review

Transcript:

[00:00] [music]

Moderator: [01:08] Hello, and welcome to the David Bloomberg BNA webinar titled "Five Things Every Legal Pro Should Know About Managed Review." I have just a couple of announcements before we begin.

[01:15] First of all, thank you for attending today's presentation. You have multiple options for audio today. If you select the dial-in option, please make sure your phone is on mute. If you select the listen-only option, please make sure the volume on your speakers are turned up.

[01:35] Questions will not be submitted by telephone. Questions may be submitted at any time via the Q&A pod at the lower right of your screen. Simply type your question and click the icon to the right to submit. Questions will be allowed to accumulate and will be addressed towards the end of the presentation or when our speakers deem appropriate.

[01:52] You will currently see instructions on how to earn credit for attending today's webinar. If you have any questions about obtaining education credits, please email us at credits@bna.com. Instructions for downloading statistics can also be found in the files pod.

[02:18] In order to obtain CLE credits for attending today's presentation, you must answer all call in questions that are presented throughout the webinar. There is no submit button for most of those questions.

[02:15] If you would like to receive CLE credit for attending today's presentation, please make sure to answer the call in questions at the end of the program accurately which will ask you for your jurisdiction where you wish to receive credit and your corresponding bar number.

[02:34] Statistics will be ready to download in your classroom at learning@bna.com in the "My Certificate" [inaudible] within one business day of the presentation.

[02:25] A copy of the presentation you are about to see as well as an additional handout are available for you to download from the top part to the right of your screen. It also provides with takeaway reference material and detail on obtaining CLE credits. We would like to thank FTI consultants for sponsoring today's program.

[02:44] FTI Technology is a practice of FTI Consulting that is dedicated to helping organizations protect and enhance enterprise value in an increasingly complex legal, regulatory and economic environment. FTI clients rely on their software, services, and expertise for matters ranging from internal investigations to large-scale litigation with global e-discovery requirements.

[03:13] I would now like to turn the presentation over to Angela Naval from FTI Consulting.

Angela Naval: [03:20] Thank you. Hello, and I am happy to introduce you to today's agenda and presenters. Welcome to "Five Things Every Legal Pro Should Know about Managed Review."

[03:35] On our presentation today we'll start with an overview of Global Data Privacy laws and move to a discussion on regulatory responses, technology assisted review, collaboration and communication with your team and how to define and measure quality.

[03:52] Finally, we'll conclude by answering questions from you, our audience using the Q&A tool on your console. Additionally, throughout the presentation, you'll notice several pop-up poll questions. We invite you to participate and answer these polls as they come through. Our speakers may or may not respond to results throughout their discussion.

[04:19] Finally, please see on the screen in front of you the following disclaimer from Jones Day, the law firm of our presenter, Keira Campbell. This disclaimer will also be provided in the downloaded PDF version of these materials. Now, let's meet today's speakers.

[04:36] Keira Campbell of Jones Day focuses her practice on antitrust and competition law. She represents clients in mergers and investigations before the Federal Trade Commission, Department of Justice, Antitrust Division, and state attorneys general.

[04:54] She has additional antitrust experience, providing counseling on distribution and trade association matters representing clients in FTC and DOJ conduct investigation, and cross-border filing. Keira practiced for a number of years in Australia, representing clients in cross-border transactions and in complex antitrust litigation. She has [inaudible] missions in both the United States and Australia.

[05:22] Welcome, Keira. Thank you so much for joining us today.

Keira Campbell: [05:26] Thanks, Angela. I am happy to be here.

Angela: [05:29] Next we have Kathryn L. Hardie, a Senior Managing Director in the FTI Technology Practice. She has 21 years of experience in litigation support, electronic discovery and channel management with particular expertise in mergers and acquisitions with an emphasis on the Hart-Scott-Rodino process and second request compliance.

[05:50] She has worked with top global law firms and Fortune 100 corporations, guiding technical needs for government compliance in response to the Federal Trade Commission and the Department of Justice subpoenas.

[06:04] Additionally, Kathryn has also been engaged in matters related to the European Commission competition investigations of several US and EEC based corporations. With that, I'd like to turn the call over to Kathryn Hardie of FTI.

[06:20] Kathryn, please go ahead.

Kathryn Hardie: [06:22] Angela, thank you so much. I know Keira and I really appreciate this introductions and summaries of what we do on a day-to-day basis. We're going to just start off here with some of the global data privacy laws that many of you may have already experienced in your lives.

[06:38] There is a continued globalization of our world. The data locations and collections of that data across borders can be extremely challenging. This heat map represents many things, but particularly I'd point out jurisdictions where there are no regulations such as China and those who have had severe privacy requirements, which are particularly notable in Europe.

[07:04] This impacts many of your decisions as you approach these test cases on how to manage multilingual reviews with alien country or not, as well as the evolving landscape of things changing in global data privacy such as Brexit that we're currently seeing.

[07:23] I can speak right now a little bit about the challenges of handling reviews that are multilingual across borders. As Angela said, I worked not only here in the US but also in Europe on cases that have spanned the globe.

[07:36] I can tell you that tackling these unique requirements can be challenging, particularly around finding qualified bilingual paralegals or attorneys. Many of whom will tell you they are qualified, but may not pass this illustrated language test that we definitely encourage you to impose. They become a critical gauging factor.

[08:00] Managing these stuffs is also critical. It requires not only an attention to the language but also culture, and we want to make sure that instructions are passed correctly on to our team, so the information is literally not lost in translation.

[08:18] As a practical result then, cross-border matters require much more thought around language and the differences in methods of managing teams, whether that's across time zones, across jurisdiction, dealing with the data, etc. This adds to creating some of the complexity around the desired consistent review decisions and outcomes for your cases.

[08:43] We definitely want to make sure that we are imposing the same thought process across all of our team, whether they are native English speakers or non-English speakers. Keira, from a legal perspective how would you describe your experience on the global arena?

Keira: [09:00] Thanks, Kathryn. The one thing that I'm looking at the polling question is I see that it's almost a split in terms of people who have had some exposure to obtaining data across borders. I think that increasingly that the convergence is going to be seen in matters whether it's a litigation matter or it's a deal matter. That's because it is not as compartmentalized reviewed anymore, increasingly international.

[09:30] What's that doing is increasingly the clients that we are serving are international. A client that maybe have its main office in the US will have offices in other countries. Unwittingly, as part of a request for documents, they're either in a litigation or a agency investigation.

[09:51] You may request document from an overseas entity, and you're just moving forth in doing that and thinking I've got to get these documents produced or whatever the case may be, you can unwittingly breach the data privacy laws of the various countries.

[10:08] As one of the takeaways from today, I think it's why we started off with that because it is something that needs to be considered from the outset. To ask yourself, "Is this deal, this litigation, is it going to involve some documents from an overseas jurisdiction?"

[10:26] If the answer is yes, at that point I would pause and reach out to a data privacy expert. I, myself, have probably enough knowledge to be dangerous on this, but I would never, consulting an expert, start to proceed in how I would go about addressing that data privacy. I suspect that for many people whose practice is primarily regulatory or litigation that that might be the same reaction.

[10:56] Once you've reached out or you've recognized that there may be a need to collect data across borders, I think that your data privacy expert will be able to guide you through what you should do or what you couldn't do.

[11:10] But a few different questions to think about asking them, and I'd like to talk more pragmatically about, how would I approach this is, are individual employee consents required to transfer data? Do those consents need to be in email? Does someone actually need to sign a consent?

[11:32] What's the language required in those consents? There's very often specific language that needs to be included for those consents to be valid. Then is there a particular national agency for privacy that needs to be notified, which for some European member states is required?

[11:54] Then importantly, especially for anybody who's dialing in as a lawyer, I've got to make sure I'm maintaining my privilege and we could have a whole another session on European versus US privilege. But one thing there to think about is in seeking those consents make sure that you are not waiving any privilege.

[12:16] Then also think about, are there any existing company client IT policies that might preempt the need to obtain these consents? These are all questions that you need to be asked if you're answering yes to that first question, "Does this involve the collection of data from overseas?"

[12:36] I would also continually ask those questions if I've done a deal one year and then I'm coming back a year or two later, and I'm going to be seeking data or documents from an overseas jurisdiction. I would never assume that the laws have stayed the same in all of those questions or those answers to the questions I discussed are the same.

[13:02] From nothing else, Brexit shows there's a lot of change in this area. Even the EU framework is constantly evolving and expanding. It moves from safe harbors to frameworks, to a privacy shield.

[13:20] This is an area that's really moving. Even for jurisdictions that have a current privacy framework, to check in and make sure that that's still applicable, though. Back to your data privacy expert. Then for countries where there may not have previously been a data privacy issue, to not assume that that's still the case.

[13:46] I would also, wanting to be looking at the different locations of employees. Because your client may have an overseas office does not necessarily mean that the employees are all working from that office.

[14:01] Often it may be the case with teleporting that in one employee, part of that entity may be located in Italy. Maybe another one's located in France. Don't assume that the same rules are applying there, even if they're all part of the European Union.

[14:18] What does all that look like practically in terms of collecting documents? How I like to approach it is because this does involve more hurdles or gates to clear, I almost approach the collection of XUS, if I could say, documents and data as a kind of a separate project from the collection of US-based documents and data.

[14:43] That's because of all these questions. I want to make sure I've ticked all my boxes and I'm not missing anything. Of course, the collections and processing are related to the same objective of whatever that may be, producing to a court or an agency.

[15:05] I think it's served well to segment them and to alert your vendor, and we'll get to talk about communicating more specifically later, but to communicate to them that there are some people who are located outside of the US, and to put everybody on an alert that that data may need to be dealt with differently. There may be some hurdles that need to be cleared before that data can be looked at and processed and reviewed.

[15:38] Continuing on from Kathryn's call-out about translations, there's absolutely a kind of a flowing effect. This, obviously, it's not tied to data privacy. You can have translations even when you're not looking at data privacy, but it often does arise that if you're collecting from outside the US there will need to be some translations.

[15:59] I would say as practically dealing with those is, again, buffering a little bit more time there as the review and the processing. I'm often surprised at how difficult it is to find, as Kathryn had talked about, quality reviewers and people who are actually really versed in a foreign language, and I never want to just rely upon one reviewer who has a particular language skill.

[16:28] That, for me, is putting all my eggs in one basket. I want to have the ability to QC each other, especially where those documents may be ones that are critical to the case. Maybe it is that that's actually the senior management's documents are in a foreign language.

[16:45] I would talk about how, again, with your vendors, where are your people located? How many are there? Are they going to be there for the entire role, and just be focused on that and allow some extra time for that?

[16:59] We'll talk about how the government agencies may also view translations a little bit later. But the one point I wanted to end on on privacy is not to forget about it. Think about it first, but also come back to it at the end, because often what happens once a litigation or a transaction is over is everybody's just so relieved to have finally made it through that process that they just forget about the data or the documents.

[17:30] But the privacy laws will often and should prompt you to go back to who those documents were provided to and to have them concerned, particularly with the agencies, that that data has been destroyed, and to relay that back to the client.

[17:49] That should be the same also for your managed review provider. You don't want to be unnecessarily holding data that's subject to privacy requirements for longer than is necessary. I think it tells you a good clean-up habit to get into.

[18:05] I think it's something that is overlooked, but it's good with privacy to think about at the beginning and at the end. I was specifically [inaudible] that for the DOJ if you would prefer to have the documents returned to you rather than destroyed.

[18:22] That's actually something that you need to specify at the time you're providing the documents to them. In terms of dealing with regulators and their responses, that's going to be covered on our next slide in more detail.

Kathryn: [18:37] Great. There are some incredibly good points. I hope you do elaborate a little bit about the agency differences and their views towards translations. I know that there's been, on many cases, like active negotiations about what is acceptable and not acceptable. To that point, the legal teams have to request for information from these agencies.

[19:00] What are some of these practical things that we should know about these agencies? First of all, they're not all created equal. Keira alluded to that across these various organizations. There's a different use of how data is managed and how data is presented.

[19:16] I would say, and Keira elaborated on this from a legal perspective, how you walk into your negotiation would be important for you to know what their views are towards using certain types of technology. The abilities, certain types of workflows, the decisions around these, the software techniques has to be considered initially.

[19:38] I know when I work with my clients and somebody says to me I have a DOJ subpoena or FTC subpoena, I immediately start thinking of the differences between these agencies, one of them, the European Commission or Canada. We'll talk a little bit about the agencies here. We can speak from our experience that technology-assisted review it's not widely used in the Competition Bureau of Canada nor the European Commission.

[20:05] They're still growing into those types of technologies. However, it is widely used. This is FTC and the DOJ. Keira can talk a little bit about some of the differences in approach that those agencies have.

[20:20] To that point, Keira, can you elaborate a little bit on your experience with the agencies and talk a little bit about what you've noticed some of the more challenging differences and processes might be, and what you feel folks on this phone might want to be most aware of?

Keira: [20:38] Sure. One of the things I will say initially is -- I'm going to focus on the DOJ and the FTC here -- they are broadly similar. A lot of the variety that you're going to receive is going to be based upon the staff that you have been appointed to the matter of the litigation that you are working on.

[21:07] That's going to apply even within the DOJ and within the FTC. They are people just like us. If they have had more experience with a certain type of review or a certain type of technology-assisted review, they're going to bring that with them. That's going to affect how many questions are asked of your processes or not.

[21:29] Maybe you're really just ticking the box in terms of the information that they're going to want. They're not going to be a lot more follow-up. That's definitely something, as an asset, that you need. I'd say, on balance, they aren't going to be similar.

[21:46] Before we get to some of the specific similarities and nuances that they will apply in terms of producing materials...I'm going to focus on producing materials in response to a second request. That's primarily what my interactions have been. They have extended to subpoenas, civil investigative demand. I'm going to largely talk to a second request.

[22:12] Before I get that, I wanted to mention something which, again, I hope is this really kind of a practical pragmatic kit that can often get lost before you get to the second request or production stage, but which heavily affects that second request. That is document preservation, also known as a litigation hold depending upon what type of matter it is.

[22:36] If you're in a contested matter or any matter that's subject to review, that your client is the subject of an investigation before the DOJ and the FTC, it's likely that your client is going to receive a notice requiring them to maintain documents related to the matter, related to...If it says it's an [inaudible] of matter, the competition concerns are, the different elements of the transaction.

[23:05] I will say that actually, the DOJ practices to tell you to expect that preservation will be maintained even if you do not receive a preservation notice. If your client is the subject of an investigation, the DOJ... [inaudible] division's practices that your client should be maintaining documents.

[23:15] What does that mean? How would you implement that? I would say immediately begin tracking and documenting. That's going to be a scene I'm going to come back to several times.

[23:45] At some point, as soon as that request or as soon as the subject of an investigation has been contacted by the DOJ or the FTC to implement that preservation notice to record what individuals are going to be covered by the document hold or the litigation hold.

[24:05] What does it actually cover? Is it going to be local email? Is it going to be server email? Is it documents on shared drive? What's going to happen to the backup tape? All these may be standing quite into the detail and more on an IT perspective. This is very critical.

[24:27] If you six months or three months down the track after an investigation has started, you do get a call for additional information or second request or subpoena. All of the materials that the government would have expected has been maintained. It's going to give rise to a lot of questions and an unnecessary sidetrack of what your real goal is here.

[24:53] I would approach this very systematic place of noting down, having a tracker about who's going to be covered, when were they covered, and what's going to happen to all of their materials. I would add, also to your benefit, to be familiar with the client's retention policies.

[25:13] Are the documents deleted? Are emails deleted? 120 days are employees able to archive their own emails. That's all something that will help you when you're interacting with the agency more generally. I think it's good background.

[25:31] You should try to be familiar with your client's systems and their databases, so that if you are anticipating there being a pull for documents and data, that you're already armed with that kind of knowledge before you go in it and talk to the DOJ or the FTC.

[25:56] For them, if I were to move from there into the second request process, I would say one of the first steps that you should take is to look at their existing co-production guidelines and model's second request that both the DOJ and the FTC have available. This is going to help in formulas to have open [inaudible] to a managed review in different types of technology-assisted review.

[26:16] The models were all fairly recent, but they're all being updated. At least, it's always an option or could be updated. The DOJ's version is from last year in June. The FTC's was in August. That's fairly relevant, but I'd always be checking to see if there's been a change and the same with the production guidelines.

[26:33] They're going to specify what's allowed and what's not. We're looking at those different models and some just kind of my general experience, I'll say that there are a few points worth mentioning that are a little bit unique between each of the FTC and the DOJ.

[26:57] Firstly, you are more going along, what I would say, maybe the traditional model of a document review using search terms which is traditional, but I think is still a very frequent way to review and produce documents.

[27:16] If you're using search terms, the FTC encourages you to provide these, but they don't require them to be provided. If you do provide them, they are open to comment on them, but they will not tell you that those terms are going to be sufficient.

[27:37] They may just give you some guidance on it if they think that there's a certain tweak to these terms or the other term. In contrast, the DOJ requires search terms if they are going to be used. They require notification of what those terms will be in Boolean strings and exactly what they're going to look like.

[27:54] As I started off with, the extent to which you receive comments on those is going to be influenced by the staff on the matter. You may spend several weeks. You may feel like talking about search terms and talking about tweaks to the search terms with the staff so that they're comfortable with the search terms you've used.

[28:15] You may send off the notice and then hear nothing further from it. It can vary quite a bit but it is something that, if you are using search terms with the [inaudible] divisions, the DOJ, you only need to provide those. Moving to the technology core review which we're going to talk a little bit more about, I'll just kind of mention some points there. However, points, we'll get more specific.

[28:45] If you were going to use any type of technological-assisted review with the DOJ, they will require a detailed description of those methods. Again, this is set out in their model second request. It's something that they will want to have, almost like a little submission on how the software, all the models that you're going to use will work.

[29:10] They may talk to you about that, again, depends upon the staff. The FTC asked to be contacted if a form of TAR is going to be used. Another kind of distinction between the two in terms of responding to a second request is privilege logs are slightly different.

[29:32] The FTC allows a partial privilege log, which is as a much more distinct privilege log where certain concerns can be identified and the number of documents that are privileged for that particular person. Whereas the DOJ will want a complete privilege log for each document that is going to privilege over the time, date, and the description of the claims of privilege.

[29:55] Increasingly, they are very closely reviewing privilege logs and the descriptions accorded to them particularly within house council. They're definitely very far in their review of the privilege log. We'll talk a little bit about more privilege and how...

[30:15] I'm going to approach that a bit later in terms of quality, but definitely something I'm sure we're all getting conscious of is privilege and being careful to let any privilege log. Getting back to translation, one of the differences...This is again, can be [inaudible] variety in translations and the approaches.

[30:37] Both agencies require that any foreign language documents are translated, but there is some scope in terms of what that means. Does it mean that you can use a machine translation, which obviously is quicker and is very cost effective? But they're not perfect. It's a machine translation.

[31:00] If that route is accepted, there may often be a cap on the number of documents that are produced that are subject to machine translation and whether you need to monitor that. Let's say maybe it's actually a percentage. A percentage of your total production, no more than 10 percent could be subject to a machine translation.

[31:19] That may be something that you need to see monitored, but you're comfortable committing to that because you think ultimately, the translation's [inaudible] .

[31:28] If they don't want a machine translation, it needs to be a human translation, then as we talked about at the beginning, that's something that is going to require seeking out appropriately trained people. It's something that is going to add a lot more time into your review, and ultimately your production.

[31:51] With all of that in mind and those tweaks between the different agencies, what's the best review workflow? What's the strategy? I'm just going to offer some questions and thoughts to think about. Often, it may feel like you just get the docs and you just start reviewing it. I think it's definitely better to think a little bit about what types of documents are going to be important.

[32:21] Are they going to be a particular file type? Maybe it happens that PowerPoints are going to be really, really interesting. Or maybe there's a section of documents. Maybe it's going to be board reports that are more interesting. Think about what are the types of documents that I really want to pay attention to, rather than just working out whatever process of review you use, search terms or TAR.

[32:47] Beyond that, beyond just thinking about, "Let's get these things out there and let's get the responses," what is substance here? Think about what are the types of documents? Also, what's going to be important, is that going to vary between the custodians?

[33:04] Usually, typically, that is the case. There is going to be a core group of individuals that are going to have more familiarity and more high-level documents than others. How does that affect your priorities for a review? What are they?

[33:21] Do you want to have certain custodians placed earlier in a production or do you want them later? Do you want them in the middle? Just generally think about what is your time frame? What is your priority? When do you want to say, "I'm done with everything?" Or do you want to just keep rolling materials in?

[33:37] Finally, this is one topic we'll talk about. What's your approach to quality control? I'm going to turn to the next slide and say all of those questions I asked should be discussed and really engaged with your vendor. You should be talking to them and this is kind of a third key point is communication and collaboration.

[34:06] You are obviously going to have more familiarity and more of an instinct as to how to answer those questions and what's best. To be really utilizing in getting the best out of your vendor, I think you should talk to them. What are your goals? What are your priorities? How can they help you achieve those?

[34:22] Be really clear on, "I need everything produced by this date," and, "Let's work to do what we need to do and let's talk about how we can compromise," and just be very clear on the expectations. To get to that, I found that there's several more pragmatic ways to help with these communications.

[34:44] I think that once you have engaged, or very soon after, is to start having regular calls with your vendor. It may be daily. It may be three times a week, once a week, or however, it is needed to depend upon the priority and review time period. That is something that I found is incredibly useful.

[35:11] Those calls have an agenda. I know Kathryn will also comment on that, but that to me is something that really helps me keep track of where we are, how things are going, and to talk about if any issues have arisen. That's with the project team.

[35:29] For the review team, if you are using...Typically, you will be using some type of assisted reviewers, to be clear and to give them some background materials. Prepare an overview presentation with them. Talk to them about that as part of an initial kickoff. Give them as much background as you can, but without overwhelming them.

[35:57] Something they can also keep later and will refer back to will include a presentation that will describe the particular litigation or the case, some documents that you think are useful or significant, so they can understand what it is that they can get a better knowledge about the matter.

[36:18] Give them your instructions on privilege, what types of documents are going to be subject to different kinds of privilege. Give them a current attorney list and continually update that attorney list. As everyone is aware, the knowledge and the understanding of a case is going to evolve. There may be other counsel that is looking at a product liability claim. Maybe it's unrelated to your case, but it comes up as part of the review.

[36:52] That also ties into my other point, which is just about feedback and telling both the project team and the review team what is good, what's not good, and to be really open about that. One way for a review team I found that can occur is through decision logs, through privilege logs, documents that attract different interpretations or views on whether something's responsive or privileged or not.

[37:23] To give them that and to have...Maybe it's a weekly call. Maybe it's twice weekly. Call with them to check in and see how they're going. All of that, and it should be iterative. It's all about not only focusing on what's relevant but the substance, what is going to help you in the long run or what you need to know about.

[37:46] I also think another important element that I referred to earlier is if you have all these different documents, you can really then refer back to your processes. If say something comes up later on, six months down the track, you can refer back to that. Here, I really rely upon my vendor to document certain things, and I'll document other things.

[38:10] If I would give a specific example, let's say a custodian interview or a data collection. At this point, as I've set the scene, I've been talking with the agency or opposing counsel, whoever it is that you're discussing that you may produce documents from.

[38:30] You told them the role of these people, what's their responsibilities, who their predecessors were, how long they've been there. You've gone through all of this and now you're at the point where you've got the custodian, you're meeting with them, and you're going to collect some documents from them.

[38:46] You think, "Let's just do this. Let's just get the materials and let's go. I've done the hard part now. Let's get the docs." I think it is really something that you need to be clear...I say we, I'm really thinking of my vendor here, needs to document clearly what was taken, what was not taken.

[39:08] Was this [inaudible] taken? Was this document not taken? Were there some time periods that we did not take? Were there hard copy documents? Were there thumb drives? This needs to be captured, so that if there's a question later about the extent of a collection, you can go back and say very clearly, "That was not taken," or, "That was included."

[39:33] It's particularly so where if you're in a transaction, or it ends up being a follow-on litigation, or you end up having to justify this later and you want to use an earlier production, you're very clear on what that production contains, what it was collected from.

[39:50] You're going to save a lot of money, a lot of time, and I think a lot of good will if you can say, "No, we're good here. We've covered that when we collected it back six months ago. Here you go. I know what we've got, and that includes this material."

[40:02] I really talk about...Communication is so critical and then the follow on from that is having somebody, where necessary, document that communication or document those decisions for later on.

Kathryn: [40:25] I could not agree with you more, Keira, on so many of those different points that you made, particularly from the lawyers and the attorneys driving the negotiations in a lot of these cases. Those of us in the support capacity really are the engine, the underbelly of what these teams are trying to do.

[40:45] Information into our organization is just as critical to make sure that engine is working smoothly. It really does become a very well-rehearsed dance of information.

[40:57] Oftentimes in our own organization, we offer up space inside of our review facilities, to allow our outside counsel to come in and spend time face to face with our attorneys to help make sure we're driving home that message around being a partner to our clients as opposed to somebody separated from them, making independent decisions.

[41:19] To that, Keira just showed a few critical points to remember, which is really, definitely understand what roles people have. Make sure that you are clear if you are driving the case, what your support provider is delivering to you in the way of personnel, managers, understanding what roles they have, what role you have, how the communication is going to flow.

[41:47] If that provider doesn't have a system, try to set up a methodology whereby you're receiving the type of reports that you need to have, whether it's a simple review or whether it's more complex and ties into load balancing what's needed in technology-assisted reviews.

[42:04] All of this will result in things that make counsel very happy, which is efficiency and lower costs. If we're able to avoid, or see down the road, errors that may occur, we're able to then cut those off at the pass and apply solutions to be able to assure we're getting through documents in a timely fashion, utilizing less labor.

[42:30] We don't want to have to repeat phases or steps to be able to result in the types of things Keira or her teams need as well, or any of you may. When we talk a little bit about collaboration and communication, I think these are the arteries that run the heart of the matter.

[42:50] If we don't have a good flow of information, we just simply cannot function properly to make sure that we're pushing these mounds of data through the systems for you.

[43:01] If you think about it for just a moment, and you think about the average type of data collection, particularly on a second request nowadays, it's approximately 15 to 20 gigabytes per custodian.

[43:12] If you think of a gigabyte being about 7,000 documents per custodian, and you multiply that to an average case of potentially 40 custodians, we're talking about millions of documents that have to be processed and pushed through a system.

[43:29] The decisions made...Always keeping in mind the quality and, to Keira's point, tracking that system completely along the way to the final production. Again, to the point of ending up in litigation, or if you're starting in litigation, that you are tracking well.

[43:45] You are able to prove that your process is working, and you're able to go back, whether the courts or opposing counsel, with a methodology that is defensible. We want to remember that, all the time, about our defensibility.

Keira: [43:58] I'm also saying [inaudible] how do you start to tackle that huge volume of documents we're now constantly encountering, even before we get to predictive coding and different types of models which we'll come to?

[44:18] Surely, I think there's some other types of ways...This is, again, just riffing on communication you can talk about with your vendor. Different types of filtering to reduce the volume down. We can touch upon search terms. Obviously, there's a variety of other ways.

[44:34] It could be date filtering. It could be eliminating certain email domains, or actually focusing on certain email domains. Maybe there's a particular person who was more loose in their documents than others, and you just want to hone in on that person to look not only for relevance but substance.

[44:55] One thing that you want to talk about is what are the different options to look at the data? I find it's really helpful to get a macro review of the data in terms of how many documents per month for a particular custodian were there?

[45:11] I can start to drill down if I'm more interested in that, but to break away at the task ahead, of course, you can eliminate spam. There's different analytics we'll also get to, but the last point is, in terms of dealing with the documents and avoiding duplication, you could decide to use issue notes.

[45:37] One thing we think about with issue notes, that is where as part of the review platform, you would like your team to identify if the document may be hot, or if it's significant, or if it's relevant to a particular claim or not, is to not go overboard to the extent that there are multiple different buttons that people can click. It's going to slow down the review, so it's a judgment call about how many you need.

[46:04] Maybe you need two. Maybe you need more. It's something to consider. Trying to be efficient and avoiding a later review may actually slow down the review. That's some things to think about. We may now want to turn to what is often talked about, which is -- and we mentioned this multiple times -- technology assisted review.

Kathryn: [46:28] Technology-assisted review can mean many things to many people. I'm going to quickly touch on some key components of using technology-assisted review, or TAR, or predictive coding in some of the interfaces that you may see in your products that you're using today.

[46:52] From what I see, there's a few different ways that you can go about using different types of software platforms. First, the various parts of any platform industry can be mixed up and matched up. What I mean by this is we can utilize data visualization tools to pull from document content and make first line review faster, more efficient, less error prone.

[47:15] This alone can reduce your costs and result in better results. It's a great way to manage a first class review. You can go one more level down in sophistication and include such features as data mining. In this functionality, you can actually see millions of documents and their contents at the macro level, allowing you to hone in on more critical language to your matter.

[47:40] This data can then be identified by a custodian and drawn back into that mapping feature for faster review and enabling the legal teams to explore information more efficiently. You can also use pivot tables.

[47:54] Pivot tables enable you to look at intersections of language, such as search terms and reference to time spans or custodians of where this information intersects and has commonality. Pivot tables are just another way to explore the data in a three-dimensional manner. I know Keira will talk a little bit about how she uses her pivot tables.

[48:17] Lastly, we've all heard about that thing called predictive coding. It's a classified document which will most likely be relevant, or irrelevant for that matter. All of these interfaces can be used alone or in tandem. You can design a unique workflow for your matters.

[48:38] Whether it's for a second request, compliance, depo prep, litigation, an exploration, an internal investigation, we have seen many corporate counsels use this for investigating data for employee issues as well.

[48:58] Technology-assisted review is not just one thing. You can think of it as many ways of using a faster platform to critically organize your data how you want to see your outcomes resolved. The last thing you want to think of this is as a black box that makes decisions for you.

[49:17] We talked about collaboration and information and communication flow. It's critical that you are engaged in the outcomes of what the technology will provide to you. It is back to that well-rehearsed dance of information of how the results of any technology assisted review will be effective for you or not.

[49:39] Keira, I talked a little bit about some of these methods, but I know you've really been in the gust of them. Maybe you can touch on a few highlights of how you use some of these interfaces in your cases, and what the best results have been from those.

Keira: [49:56] Definitely, and I just want to reiterate your point, Kathryn. It's really got to be dependent upon the case. It shouldn't just be running full steam into using predictive coding or other review forms, because it's really dependent.

[50:12] There's a few things that I'm going to ask, or I always ask before a review approach is determined. No one doubts that predictive coding is very efficient. It's made review and production of documents more feasible, but it's not going to be right for every case.

[50:34] Things to ask is what's the volume of documents expected? That's going to drive whether you should use predictive coding. If you're only looking at maybe 100,000 documents, predictive coding may not be worth all of the effort.

[50:51] What's going to be the scope of the request and the responses sought? Are they going to be broad and varying or are they narrow and related? If it's the latter, predictive coding, again, it could be better in some instances. It could be not as good.

[51:08] What sort of timing? When do you need all the materials produced by? A longer time frame or a shorter time frame? What types of documents? If you're getting lots of hard copy documents, predictive coding is not going to be the best way to review and predict on those documents. Often it's the case that they don't have the same feasibility of review as soft documents, as native materials.

[51:33] I'm conscious of the time. I'm just going to call out a few things that, if you are decided to elect predictive coding and you're before the FTC or the DOJ, there are definitely some considerations and some specificity about predictive coding. One of those is who's actually going to train the computer. Who's going to train the algorithm or model or whatever you want to call it.

[51:56] There is definitely a view of the agencies that they want law firm attorneys, that they want subject matter experts to be training the model. That is what will ultimately be the decision maker for determining something as responsive or not. There's less of that appetite for using contract attorneys for that task. It could be a mix of both, but that's something to think about.

[52:21] Data, the pure data cannot be subject to predictive coding, certain types of files. Sometimes some Excel, there's even a change on that. If you are [inaudible] before the agencies they're going to want to see a non-responsive sample of an output from the predictive coding, and there's going to be a certain matrix they're going to want to see before they let the computer run wild or take over the document.

[52:54] They're all things that an experienced vendor will be familiar with and that you will talk about with them. Talking about matrix, you may just want to turn to quality and discuss that briefly, so the next slide.

Kathryn: [53:08] Absolutely, folks. In a lot of ways quality should be what we think about across this entire process because, after all, without that we're not serving our clients correctly. It is measured on many fronts.

[53:23] It's about how we select review managers and their background that goes through to our reviewers. It's based on a team of professionals that constantly recruit, has measure for outcomes, and many of which we've spoken about already. Whether that's language testing, whether that's background testing, etc., or background searches.

[53:46] On each of these matters, all of the items we discussed in this presentation from data privacy, multilingual reviews, knowledge of the agencies, likes and dislikes, proper use of software tools, directly impact the quality of your outcome. I think we've emphasized that a lot.

[54:06] I think when we talk about quality, it really does circle around the ability to communicate properly, explain properly products and outcomes and, in a lot of ways, to try to anticipate what our team and our attorneys and our clients need. It's that innate nature or ability to anticipate problems and look down the track that I believe really ensures a high-quality product.

[54:33] With that, Keira, I'll turn a little bit to how law firms manage quality. Then we will wrap it.

Keira: [54:40] I'll just comment quickly so we can move to any questions. Would say quality should be iterative. It shouldn't be something that's left to the end.

[54:48] I say you want to be having those daily calls and there's certain metrics I want to see. How many docs are being reviewed? What's the pace? How many significant documents were there? Giving feedback, asking whether your vendor has engaged in the particular agency that you're before.

[55:10] I think there's something that needs to be considered the entire time. What's the production? Is that in terms of the requirements for the agency? I think quality is huge. I think if you have good communication, good feedback, you can often really work to get quality.

[55:30] I think I might just leave it there. Maybe we'll open it for questions.

Angela: [55:36] Very good. Thank you, Keira Campbell of Jones Day and Catherine Hardy or FTI. As a reminder to our audience, you are invited to submit questions on the Q&A tool on your console. We have time for just one or two questions. We'll start with the first question that is for Keira.

[55:55] Keira, the question reads, "Are there any general best practices on handling data that resides in China?"

Keira: [56:04] That's a really good question. I think that that gets back to where we started. I would say that this is constantly evolving actually. Just from an annual trust perspective, there was a decision last month from one of the three anti-trust agencies in China that as part of its evidence for finding a prohibition of certain anti-trust acts used web chats, personal web chat communications, as part of their evidence.

[56:39] That raises questions on is that protected? Is that subject to privacy? I think it's something that's evolving and it's definitely becoming more to the foreground. It's percolating and people are thinking about it.

[56:52] I'm not aware of best practices. What I would say is, as I said at the beginning, I'd definitely watch the space and go and check with a data privacy expert on managing data in China.

Angela: [57:14] Thank you, Keira. We have time for one more question. Again, we invite you to submit your questions through the Q&A tool. If we are not able to your question, you may receive an email response.

[57:28] The next question is also for Keira. "Keira, which agency is the most interventionist?"

Keira: [57:36] That's a really good question. Going back to metrics, what metric? I would say in terms of looking at, say, from a production of documents and responding to calls for materials, that the DOJ and the FTC really lead in terms of approaches to predictive coding and familiarity with predictive coding.

[58:06] I think the Canadian Competition Bureau -- it's not yet there and the same with the European Commission. But they are definitely aware of the technology. They're just a little bit more based in the traditional model. One country and agency we haven't talked about is the UK. I think that's another area where I'm sure there's a lot of review before the Competition and Markets Authority there.

[58:42] They haven't yet got a policy on using technology for assisted review. But there was earlier this year a decision in the High Court there, the Pyro case, that actually talked about predictive coding and came out in favor of using predictive coding.

[59:06] I think that that's something again -- keep the lookout -- that we would see there's going to be a continued uptake of predictive coding and TAR, which is hopefully something that made this presentation timely and attractive, is that maybe you haven't used or you're not as familiar with it, but it's definitely good to be familiar with it as it's continuing to expand.

Angela: [59:27] Thank you, Keira. Definitely. This concludes today's presentation. I believe we have one more housekeeping note from our producer with Bloomberg. But before we get to that I'd like to thank again Keira Campbell of Jones, Day, and Kathryn Hardie of FTI. Thank you for presenting. Thank you to our audience for joining us today.

Moderator: [59:54] Great, thank you so much, Angela. You will currently see instructions on how to earn credits for attending today's webinar. Please go to learning.bna.com to retrieve your certificate. Daily certificates may take four to six weeks to appear depending on the jurisdiction requested.

[60:08] If you have any questions about continuing education credits, please email us at credits@bna.com. Finally, I once again thank our speakers for excellent presentations, FTI Consulting for sponsoring today's program, and all of you for your participation in today's webinar. Thanks to everyone, and enjoy the rest of your day.

[60:27] [silence]