Final Report - AOC-ODR Program Evaluation & Recommendations 2-2022 (2)

PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 1 of 72 PRAXIS Management Solutions, LLC Submitted 7 February 2022 Final Report to the AOC ONLINE DISPUTE RESOLUTION PROGRAM EVALUATION AND RECOMMENDATIONS Respectfully Submitted on 8 February 2022 PRAXIS Management Solutions, LLC Steven G. Meilleur, Ph.D., SPHR Peter C. Meilleur, SHRM-SCP 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 2 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 NEW MEXICO ADMINISTRATIVE OFFICE OF THE COURTS (AOC) ONLINE DISPUTE RESOLUTION PROGRAM EVALUATION AND RECOMMENDATIONS This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is provided with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services. If legal advice or other expert assistance is required, the services of a competent professional should be sought. © 2022 PRAXIS Management Solutions, LLC All rights reserved. This publication may not be reproduced for external use in part or in whole by any process without express written permission from PRAXIS Management Solutions. Questions or comments about this publication? Contact: Steven G. Meilleur, Ph.D., SPHR or Peter C. Meilleur, SHRM-SCP PRAXIS Management Solutions, LLC 316 Encantado Ridge Court, NE Rio Rancho, New Mexico 87124 Ph (505) 401-0942 [email protected] PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 3 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ONLINE DISPUTE RESOLUTION: ODR PROGRAM EVALUATION INTRODUCTION This report summarizes the results of a program evaluation study of the N.M. Administrative Office of the Courts’ Online Dispute Resolution Program (hereinafter ODR). Purpose of the Study The purpose of this study is to conduct a program evaluation of the Online Dispute Resolution (ODR) program offered through the NM Administrative Office of the Courts (AOC). The specific focus of the study was to assess the effectiveness of the Modria-Tyler Technologies (Modria) online program in handling and settling “Debt and Money Due” cases referred by the New Mexico courts. Beyond the assessment of the Modria program, the researchers sought to evaluate the and to make recommendations for improvement of the overall ODR program operated by the AOC. It is hoped that the results of this study will help the AOC to improve the quality and effectiveness of the ODR programs and will help the AOC to improve access to justice for New Mexico’s citizens. What is “Online Dispute Resolution (ODR)”? For purposes of this paper, the term “ODR” can be used in two ways, each of which hopefully is clear from in use from the context: 1. In one use, the term “ODR” refers to the Tyler-Modria online platform used in the AOC pilot program for handling and settling “debt and money due” cases, described in the Modria-New Mexico Courts poster (Appendix 6), and is a primary focus of this program evaluation study. 2. The term, “ODR” can also refer to the broader application of technology-based methods in which the disputants can come together to resolve their disputes or cases. The National Center for State Courts (NCSC ODR Project, n.d.) defines “ODR” as (highlights ours): Court-related Online Dispute Resolution (ODR) is a public facing digital space in which parties can convene to resolve their dispute or case. Three essential components differentiate court-related ODR from other forms of technology￾supported dispute resolution: The first is that the program operates exclusively online. In contrast to other court programs that provide an online interface with which to accomplish discrete tasks (e.g., e-filing, video hearings), ODR users do not otherwise interact with the court for traditional in-court procedures or events. The second is that the program is explicitly designed to assist litigants in resolving their dispute or case, rather than a technology platform to support judicial or court staff decision￾making. Dispute resolution inherently includes the potential to challenge the validity of claims or to raise affirmative defenses; court-related ODR is not merely a platform for defendants to negotiate a payment schedule to satisfy debts. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 4 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 Third, the program is hosted or supported by the judicial branch. It is not a form of private ADR, but instead integrates and extends dispute resolution services offered by the judicial branch into digital space to serve citizens efficiently, effectively, transparently, and fairly. This definition of court-related ODR can encompass a variety of methods and tools to assist in dispute resolution. It can provide dispute resolution services without necessarily filing a formal complaint. It can support a variety of decision-making aids including discovery exchange; direct party-to-party settlement negotiations; synchronous or asynchronous mediation support; and technology-supported adjudication. When litigants successfully resolve their dispute, the program can populate standard settlement agreement forms that can be automatically filed with the court, if needed to dispose the case. If the litigants are unsuccessful, the program can also provide a seamless entry to the court’s traditional dispute resolution by automatically populating and filing necessary court forms. The design and implementation of court-related ODR programs should not diminish due process or access to justice for program users. The majority of the data collected and presented in this program evaluation report pertains to the Tyler-Modria online platform used in the AOC pilot program for handling and settling “debt and money due” cases. However, many of the respondent comments addressed the importance of building on the lessons learned from the Modria platform to develop more effective ODR services, using broader online dispute resolution methods to support ADR efforts. A majority of the recommendations in this program evaluation report are in support of this vision of moving forward to use ODR as a means to improve access to justice for all New Mexicans. The New Mexico Courts-AOC ADR Program The New Mexico Courts system, through the Administrative Office of the Courts (AOC), has long been a widely recognized leader in supporting Alternative Dispute Resolution (ADR) as an alternative to ligitation is resolving disputes in New Mexico. A featured page on the New Mexico Courts website is “ADR: Paths to Settlement.” The site describes ADR as “a wide variety of methods used to reduce or eliminate the problems associated with disagreements. These methods include mediation, facilitation, negotiation, settlement conferences, arbitration, fact-finding, and early neutral evaluation.” The ”ADR: Paths to Settlement” website also features a page, “What is Online Dispute Resolution (ODR)?, which provides in-depth information about the Tyler-Modria ODR pilot project describing it as “Online Dispute Resolution (ODR) is a free tool provided by the New Mexico Judiciary to resolve a Debt & Money Due cases online.” In addition to the ODR program, other ADR services offered through the New Mexico Courts include: ● Mediation - In mediation, the people who share the dispute work together to decide how to be resolve the situation. This process is guided by a mediator, who helps people to discuss the issues and to reach an agreement to resolve some or all of the problem. ● Arbitration - Arbitration is a formal process with strict rules about evidence and conduct. The parties to a dispute agree in writing to comply with (or “bind” themselves to) the final decision before the process begins. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 5 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Early Neutral Evaluation - A subject matter expert evaluates the strengths and weaknesses of the case and gives an opinion about the likely outcome of the case (if it is ultimately resolved by a court). The information can help the parties to better understand the strength of the available evidence and how the law applies to their situation. ● Settlement Conference - A settlement conference can be many different things, depending on the person conducting the session. The impartial person may be a retired judge, a lawyer, or someone with strong mediation and negotiation skills. ● Settlement Facilitation - A settlement facilitator is an impartial guide who helps each side to understand the strengths and weaknesses of their case and to make settlement offers. The parties may meet together or in separate rooms with the impartial facilitator going between rooms to assist both sides to reach a solution of issues. In the context of the broader definition of ODR (NCSC ODR Project, n.d.), using lessons learned from the Tyler-Modria ODR pilot project, the New Mexico Courts have a tremendous opportunity to expand ODR to support these ADR programs and services. Taking advantage of this opportunity is a key recommendation in this program evaluation report. Increasing Access to Justice, and the Potential of ODR According to the National Center for State Courts (2015), civil court dockets are composed mainly of “high-volume” cases—routine legal matters that typically never reach a judge and instead become rote transactions that court staff manage and litigants usually handle themselves. For instance, more than three-quarters of civil cases in state and local courts involve claims of $5,200 or less, primarily contract disputes such as debt collection or landlord-tenant issues. These kinds of cases, though relatively small in dollars, present significant financial and time challenges for New Mexicans. This experience was reflected in the findings of a national survey conducted by the nonprofit National Center for State Courts (NCSC), in which more than 80 percent of respondents said they want more online access to local courts, including the ability to ask for guidance from court staff rather than come to the courthouse. Existing digital technologies could provide part of the solution and increase people’s remote access to the courts (Rule, 2015). In particular, online dispute resolution is an electronic tool already in use in the private sector that helps resolve disagreements between consumers and online retailers. Research suggests that when properly planned and implemented in a court context, such innovations could make legal processes more efficient and provide a cost-effective alternative way to resolve legal issues—especially high-volume cases that clog the courts. However, more study is needed to address concerns raised by some stakeholders about whether and how well the private-sector success of ODR will translate into policy and ultimately the practice of law. This study is an attempt to contribute to that understanding, particularly as applies to New Mexico. How Does Court Online Dispute Resolution Work? ODR was developed initially in the private sector to facilitate the quick resolution of conflicts to the satisfaction of both parties. Because of its utility in quickly resolving lower-value, high-volume cases, ODR has begun to spread from the private sector to public courts. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 6 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 Although ODR is not appropriate for every legal issue, it has promise for high-volume cases involving transactional disputes, such as traffic offenses, small claims, and low-conflict family court cases. The platform enables litigants to communicate about disputes online and reach a resolution that can be authorized and enforced by the courts (Rule, 2015). In a study from the Pew Charitable Trust (2019), functions that can and should be built into court ODR systems include: ● Asynchronous communication. Individuals can access ODR at any time during or outside of court hours, leave messages for other parties, and be notified of responses. ● Legal information. The system can be built to provide legal information—without giving advice—about the procedural requirements and options available at each stage in the process to all parties to a dispute. ● Triage. ODR systems present users with different paths and provide them with the necessary information to make informed decisions, such as whether and when to request a mediator to assist with negotiations or to exit the ODR system and return to the traditional court process, each of which affects the amount of litigant and court resources required to proceed to resolution. ● Electronic document management. ODR can allow for easy sharing of materials with other parties and connect to platforms that allow participants to electronically file documents with the courts. ● Mediators. Courts can make trained mediators available to litigants via ODR systems to help them work through their disputes. ● Negotiation spaces. ODR systems can allow litigants to talk through the dispute with or without a mediator ina secure chat space. ● Document creation. The system can convert agreed-upon terms into court documents. ● Payment methods. ODR tools can allow individuals to submit agreed-upon payments through the ODR platform. What Would Successful ODR Platforms Achieve? Effective court ODR systems should deliver the following benefits (Pew Charitable Trust, 2019), which provides a basis for the effectiveness of the Modria pilot program, as well as providing quality benchmarks to be used in planning and implementation of future ODR programs by the AOC ODR office: ● Speedier resolution. People who currently wait months to get their cases on crowded court dockets would encounter fewer such unnecessary delays. The American Bar Association’s Commission on the Future of Legal Services (2016) suggests that successful court ODR would increase judicial efficiency. ● Greater engagement in the legal process. When court can be accessed via a computer or smartphone at any time, rather than only in person during business hours, both sides should be more able to actively participate in their case, with fewer no-shows and defaults. ● Increased exercising of legal rights. ODR platforms that provide users with accessible and relevant information could help ordinary people better understand their legal rights and options and help them leverage the laws and precedents that support and protect those rights. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 7 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● More fair outcomes. In the U.S. legal system, a just judgment is one that is rendered based on the law in full light of all relevant facts. ODR has the potential to improve the fairness of the civil legal system and increase the likelihood of just resolutions by reducing default rates, procedural mistakes, and other inefficiencies, providing legal information and access to key court processes, and helping to ensure that all parties to a case have an opportunity to participate. ● More efficient court processes. Several parts of the court process must be modified to fit an ODR platform, such as certain notary and reporting requirements, docket scheduling, and document delivery rules. Implementation could enable judicial leaders to streamline and improve court processes for litigants, whether online or in court. Determinants of ODR Success Court ODR is gaining popularity around the country, but some questions have surfaced about its potential effects as courts deploy systems without recognized best practices or evidence of impact, which users benefit most, or the types of cases for which these tools yield the best return on investment. Practitioners, researchers, and court leadership can work together to identify and develop needed user protections and collect and share data to inform rigorous program evaluations. Key measures and determinants of success should include: ● Access to Justice. Case filing and completion rates; representation rates; case durations; number of procedural errors. ● Cost and benefit. Court staff and online user time spent in ODR compared with traditional court cases; court time saved and diverted to complex cases; costs to taxpayers and users of implementing ODR platforms; costs to litigants of participating in ODR versus standard court processes. ● Fairness. Case outcomes when using ODR versus traditional court services; users’ perceptions of the fairness of the ODR process. (Pew Charitable Trust, 2019) It is hoped that this study will contribute to the on-going growth and improvement of ODR in New Mexico, that will meet these criteria. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 8 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 OVERVIEW OF RESEARCH APPROACH METHODOLOGY For this program evaluation study, qualitative research methods were utilized by the researchers, who used unstructured interview, focus group, and survey methodologies to gather data. The data was collected via researchers’ notes, and from respondents’ online survey written comments (See Appendix 1). Content analysis was performed on the data collected, in order to identify themes and trends as perceived by the respondent participants in the study. This section of the report provides an overview of why the methodologies were chosen, as well as the limitations of how the results of the study should be interpreted and utilized. WHAT IS QUALITATIVE RESEARCH? Qualitative research uses non-statistical methods to gain understanding about a population. In other words, you’re not dealing with the numbers you’d find in quantitative research. Qualitative research is generally defined as research that utilizes open-ended interviewing or focus groups to explore and understand the attitudes, opinions, feelings and behavior of individuals or a group of individuals. Both of these methods were used in this study. The main reason for using qualitative research methods in evaluation research is that they can provide details about human behavior, emotion, and perceptions that quantitative studies cannot match. Qualitative methods thus provide a depth of data that can help one to understand issues from a myriad of perspectives, and they can uncover issues that may not be apparent on the surface. ● Data from qualitative studies describes the qualities or characteristics of something. You cannot easily reduce these descriptions to numbers—as you can the findings from quantitative research; though you can achieve this through an encoding process. ● Qualitative research studies can provide you with details about human behavior, emotion, and personality characteristics that quantitative studies cannot match. ● Qualitative data includes information about user behaviors, needs, desires, routines, use cases, and a variety of other information that is essential in designing a product that will actually fit into a user’s life. While quantitative research requires the standardization of data collection to allow statistical comparison, qualitative research requires flexibility, allowing you to respond to user data as it emerges during a session. Thus, qualitative research usually takes the form of either some form of naturalistic observation such as ethnography or structured interviews. In this case, a researcher must observe and document behaviors, opinions, patterns, needs, pain points, and other types of information without yet fully understanding what data will be meaningful. WHEN SHOULD QUALITATIVE RESEARCH BE USED? In a nutshell, qualitative research is best used for depth, rather than breadth of information. While quantitative surveys are an outstanding medium for gathering a breadth of information regarding "How many?" or "How much?," qualitative research is the best research method for discovering underlying motivations, feelings, values, attitudes and perceptions. Though qualitative research has many applications, some of the ones we focused on in this project are listed below: PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 9 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Investigating current perceptions about the AOC-ODR program. ● Identifying strengths and areas for improvement. ● Brainstorming/Idea Generation. ● Obtaining detailed information about personal and group feelings, perceptions and opinions. ● Identifying a broader range of information, and offering the opportunity to seek clarification. RESPONDENT PERCEPTIONS IN QUALITATIVE RESEARCH It must be remembered that the research questions concentrate on asking respondents to identify their perceptions of user and implementation issues pertaining to the AOC ODR program, which according to Heil (1983) involves the interpretation of incoming information on the basis of learned norms and dimensions. Gardner (1985) defines this process as the idea of “unconscious inference,” proposing that we draw on what we have learned in the past to interpret what we perceive. Because this study was concerned with factors influencing the user experience with the implementation and utility of the AOC-ODR program, an examination of the respondents’ perceptions seemed most appropriate. Perceptions are valuable because they are indicative of potential behavior as the implementation and use of ODR transpires. As Lewin (1936) has pointed out, behavior is in response to perceptions of reality and not in response to an objective reality. While not necessarily deterministic of future action, cognitive perceptions can be suggestive of eventual activity and help improve strategic planning for adoption of innovations (Mannes, 1990) such as online dispute resolution programs. WHAT ARE THE STRENGTHS AND LIMITATIONS OF QUALITATIVE RESEARCH? Qualitative research has numerous strengths when properly conducted, but it also has limitations and weaknesses that must be understood when using the research findings. One of the most important things to understand is that data are collected from a small, purposive sample of individuals so findings cannot be generalized to a larger population. The data from focus groups, interviews, and the qualitative survey represent the perspectives and experiences of those participating in the qualitative study, and do not necessarily represent the perceptions or experiences of others in the larger applicable population. Strengths of Qualitative Research ● Issues can be examined in detail and in depth. ● Interviews are not restricted to specific questions and can be guided/redirected by the researcher in real time. ● The research framework and direction can be quickly revised as new information emerges. ● The data based on human experience that is obtained is powerful and sometimes more compelling than quantitative data. ● Subtleties and complexities about the research subjects and/or topic are discovered that are often missed by more positivistic enquiries. ● Data usually are collected from a few cases or individuals so findings cannot be generalized to a larger population. Findings can however be transferable to another setting. Limitations of Qualitative Research PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 10 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Research quality is heavily dependent on the individual skills of the researcher and more easily influenced by the researcher's personal biases and idiosyncrasies. ● Respondents are providing their perspectives, which may not accurately reflect objective truth of reality. ● The volume of data makes analysis and interpretation time consuming. ● It is sometimes not as well understood and accepted as quantitative research within the scientific community ● The researcher's presence during data gathering, which is often unavoidable in qualitative research, can affect the subjects' responses. ● Issues of anonymity and confidentiality can present problems when presenting findings ● Findings can be more difficult and time consuming to characterize in a visual way. The primary limitation of all qualitative research methods is that, unlike quantitative research, the findings are not statistically projectable to the population under study. So for this study, the findings in this report do not necessarily represent the opinions and perspectives of the entire population of Poms & Associates employees. As such, the findings and subsequent recommendations should be given careful consideration prior to taking specific action. While some of the recommendations may appropriately be adopted based on this qualitative research, it may also be appropriate, concerning certain issues, to conduct additional research prior to taking action. REASONS FOR USING FOCUS GROUPS Focus groups are a useful method to: ● discover how different groups think and feel about a topic and why they hold certain opinions ● suggest potential solutions to problems identified ● inform decision-making, strategic planning and resource allocation ● to add a human dimension to impersonal data Advantages of focus groups ● they are useful to obtain detailed information about personal and group feelings, perceptions and opinions ● they can provide a broader range of information ● they offer the opportunity to seek clarification Disadvantages of focus groups ● there can be disagreements and irrelevant discussion which distract from the main focus ● they can be hard to control and manage ● they can to tricky to analyze ● they can be difficult to encourage a range of people to participate ● some participants may find a focus group situation intimidating or off-putting; participants may feel under pressure to agree with the dominant view ● as they are self-selecting, they may not be representative of non-users. REASONS FOR USING UNSTRUCTURED, INFORMAL INTERVIEWS PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 11 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Interviews are a useful method to: ● investigate issues in an in-depth way ● discover how individuals think and feel about a topic and why they hold certain opinions ● inform decision making, strategic planning and resource allocation ● sensitive topics which people may feel uncomfortable discussing in a focus group ● add a human dimension to impersonal data Advantages of interviews ● they are useful to obtain detailed information about personal feelings, perceptions and opinions ● they allow more detailed questions to be asked ● they usually achieve a high response rate ● ambiguities can be clarified and incomplete answers followed up ● interviewees are not influenced by others in the group ● some interviewees may be less self-conscious in a one-to-one situation. Disadvantages of interviews ● they can be very time-consuming: setting up, interviewing, transcribing, analyzing, feedback, reporting ● they can be costly ● different interviewers may understand and transcribe interviews in different ways. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 12 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PROGRAM EVALUATION RESEARCH DESIGN AND METHODOLOGIES The following research methods were used to gather information and data in conducting the evaluation study over the research period: • Practice Case for Researchers to Use Modria • In-Depth Qualitative Individual Unstructured Interviews • Focus Group Research • Online Qualitative Survey • Secondary Research and Document Review These research methods and methodology is summarized below in this section. 1. PRACTICE CASE FOR RESEARCHERS TO ASSESS MODRIA: In order to gain first-hand experience with and knowledge of the Modria system, the AOC-ODR office set up a practice case for the researchers, each of whom role-played a party in a Debt and Money Due case. Even though the researchers both have high levels of experience and expertise in the use of computer-based software systems, their experience mirrored the experiences conveyed by the respondents in the interviews, focus groups, and surveys, finding the platform to be confusing and cumbersome to use. 2. IN-DEPTH QUALITATIVE INTERVIEWS: Using naturalistic inquiry, conducting in-depth qualitative interviews with key informant respondents. The initial respondents was identified with with input and guidance from the AOC ADR staff. Additional interviewees were identified using a “snowball” sampling method, thereby inserting some measure of randomization in the selection of the respondents. The researchers conducted in-depth, open-ended interviews with nine respondents representing individuals who are familiar with the AOC ODR program, including attorneys, judges, users of the ODR program, and AOC ODR staff. Marshall and Rossman (1989) recommend the in-depth interview as a method whereby “the participant’s perspective on the social phenomena of interest should unfold as the participant views it, not as the researcher views it (p. 82).” The strengths of using the in-depth interview for this study initially were that large amounts of data were collected rather quickly, a wide variety of information emerged during the interviews, and the ability to ask follow-up questions helped to clarify and expand on the participant’s responses. The use of open-ended questions was also important to this exploratory phase of the research, having a number of advantages over closed questioning techniques (Marshall and Rossman, 1989): • They permit an unlimited number of possible answers. • Respondents can answer in detail and can qualify and clarify responses. • Unanticipated findings can be discovered. • They permit adequate answers to complex issues. • They permit creativity, self-expression, and richness of detail. • They reveal a respondent’s logic, thinking process, and frame of reference. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 13 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 The respondents were selected using purposive sampling based on input from the AOC ODR professional staff, and the researchers’ prior knowledge of the ODR program. A non-probabilistic (or "purposeful") sampling technique was used to identify key individuals to participant in the interviews (Merriam, 1988). Also referred to as “elite interviewing,” (Marshall & Rossman, 1989), “elites are considered to be the influential, the prominent, and the well-informed people in an organization or community. Elites are selected for interviews on the basis of their experience in areas relevant to the research (p. 94).” All interviews were conducted via Zoom, recorded with the respondent's permission, and transcribed in whole or in part and coded, with written notes and reflections on the interview documented immediately following each interview by the researchers. All interviews were conducted at a time convenient to the participants, and in their own offices so that the settings were comfortable (Miles & Huberman, 1984). All respondents were quite open with their comments and opinions. As an indicator of the level of interest that the respondents had in the topic, all but one of the interviews lasted well over an hour, and with two interviews conducted over two sessions, each lasting over an hour. The data analysis from the interviews and focus groups (see below) provided the conceptual framework to further develop the research questions and hypotheses, and to aid in the design of the online survey questionnaire. 3. FOCUS GROUPS: Two “dual moderator” focus groups were convened (one homogenous group, one mixed audience) to explore the research question(s) and to identify key improvement needs and issues faced by the AOC ODR program. Focus Groups are generally used to gather people’s opinions, ideas, and beliefs on a certain topic, service, or product. Responses in a focus group are open ended, broad, and qualitative. They provide more depth and get closer to what people are really thinking and feeling even though their responses may be harder or even impossible to record on a scale. (Krueger, 1989; Morgan, D.L.,1989) Focus group discussion is sometimes seen as synonymous with interviews, especially the semi￾structured “one-to-one” and “group interviews” (Parker & Tritter, 2006). Similarities between these techniques relate to the tendency to uncover people's perceptions and values (e.g. Hargreaves, 1967; Lacey, 1970; Skeggs, 1997). Consequently, there are cases where authors have confused and conflated these two distinctive methods (Parker & Tritter, 2006). However, the role of the researcher and the relationship with the participants points to a fundamental difference between the two techniques (Smithson, 2000). Interviews involve a one-to￾one, qualitative and in-depth discussion where the researcher adopts the role of an “investigator.” As such, the researcher asks questions, controls the dynamics of the discussion, or engages in dialogue with a specific individual at a time, resulting in discovering the in-depth perspective of that respondent. In contrast, in a focus group discussion, researchers adopt the role of a “facilitator” or a “moderator.” In this setting, the researcher facilitates or moderates a group discussion between participants and not between the researcher and the participants. Unlike interviews, the researcher thereby takes a peripheral, rather than a centre-stage role in a focus group discussion (Krueger, 1989). As such, the richness of the responses from the group is enhanced by the “interactive PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 14 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 effect” of the participant interaction in their discussion of the focus questions (Gronkjaer, M., et. al., 2011). The first focus group was conducted with the AOC ODR professional staff, and a member of the judiciary; the second was a group of respondents selected in a similar fashion as the interview respondents discussed above. As with the interviews, the focus groups were conducted via Zoom, recorded with the respondent's permission, and transcribed in whole or in part and coded, with written notes and reflections on the interview documented immediately following each interview by the researchers. The data analysis from the interviews and focus groups also helped to provide a conceptual framework to aid in the design of the online survey questionnaire. 4. ONLINE QUALITATIVE SURVEY: An online qualitative survey process was used to broaden the research base of study by providing access to respondents from across New Mexico and from multiple audiences and users. The survey was sent out via email to who were identified with input and guidance of the AOC ODR professional staff, and contained a link to the survey questionnaire conducted on Survey Monkey. The survey focussed on identifying issues of concern and opportunities for improvement using content analysis of the data. The survey was developed by the researchers according to the recommendations of Delbecq et. al. (1975) to include a letter of introduction from Judge Jane C. Levy, Second Judicial District Court; and Josh Pando, Statewide ADR Program Manager – See Appendix 4), (2) the researcher’s email letter of invitation to participate, (3) an “agreement to participate” option, and (4) the survey questionnaire (See Appendix 3). The survey was field tested and reviewed by several of the researcher’s professional colleagues, who were asked to provide feedback and comments concerning the clarity and ease of use of the entire packet. Only minor adjustments were made to the packet based on their feedback and comment. The open-ended focus questions were: • Think about your involvement with and participation with the AOC Online Dispute Resolution Program (ODR). How effective do you think the program is today, and how do you think it can be improved? • Thinking about the Online program (Modria – Tyler Technologies) being used for Debt and Money Due cases today, based on your expert opinion, please list ten (10) things that you think should be changed in order to improve the effectiveness of the program to successfully resolve cases? • From your perspective, what four (4) things would you recommend that the AOC do to improve and possibly grow the overall use of Online Dispute Resolution in New Mexico to better provide access to justice for its citizens? • What additional comments and suggestions do you have regarding the Online Dispute Resolution program, that you think are important for the AOC to hear? Respondents were also asked to list the names and contact information (email or phone number) for up to four other people who they believed would be helpful as participants in this study. Those people identified were then sent invitations to participate in the survey. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 15 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 A total of 447 email invitations were sent out, and there were 44 individuals who participated in the survey, for a ten percent (10%) response rate. The researchers believe that a number of the emails were stuck in spam filters, which caused a lower rate of return than was anticipated. The survey responses were compiled into thematic categories using content analysis, and are presented unedited in Appendix 1. 5. SECONDARY RESEARCH AND DOCUMENT REVIEW: Throughout the research, numerous documents were also collected from individuals, planning bodies, and staff who self-identified such documents. A number of research reports and articles were obtained from various sources via the Internet, and used for reference and comparative analysis purposes. The documents which were collected and reviewed included research reports, conference proceedings, studies, legislation and public policy recommendations, along with other information pertaining to the adoption and implementation of online dispute resolution (ODR) programs and technologies. This information was used to gain background information and to aid in understanding the institutional, political, and public policy environments pertaining to the research topic. The documents also contributed to the triangulation of the data collected, thereby enhancing the generalizability and external validity of the study. Some of the findings from these documents are reported in this study, and a number of articles have been attached to this report as Appendix 8. Limitations of Research It should be noted that the generalizability and predictive utility of the study’s findings must be kept in perspective, in that the sample populations in the interviews, focus groups, and online survey phases of the study were selected using non-probabilistic methods. The interviewees and focus group members were selected using a purposive sample, and the survey respondents self-selected from a comprehensive listing of “online dispute resolution” experts, and experienced users of the AOC-ODR program. The initial pool of potential respondents was developed from lists provided by the AOC-ODR office, expert referrals, and daisy-chain referrals. As such, neither the samples for the interviews, focus groups, or online survey were drawn in a probabilistic random fashion to ensure that they were representative of the general population of people familiar with the ODR system. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 16 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 RESEARCH FINDINGS AND SUPPORTING DATA This section summarizes the findings by the research method used: 1. Practice Case for Researchers to Assess Modria 2. In-Depth Qualitative Individual Unstructured Interviews 3. Focus Group Research 4. Online Qualitative Survey 5. Secondary Research and Document Review As discussed earlier, the presentation of findings and data result from gathering qualitative, experiential data, and performing content analysis to identify themes and other issues that emerge from the data. A more completed presentation of the data is included in the Appendix. 1. Practice Case for Researchers to Use Modria On October 17, 2021, the researchers conducted an evaluation of Modria using two mock online dispute resolution sessions in which the researchers experienced the program as both plaintiff and defendant. In an effort to best simulate the experience of parties engaging in online dispute resolution, we intentionally avoided receiving any training from the staff of either the Administrative Office of the Courts or Tyler Technologies other than the video modules and other information available to the public at https://adr.nmcourts.gov/home/odr/. In our simulated end user evaluation, we were primarily concerned with looking at the effectiveness of available training, user interface and navigation, and communication options. The available training videos are descriptive in nature and give instructions for login, making offers, communication, requesting a mediator, and potential outcomes. The ideal outcome of training in any situation is to give or enhance the skills and knowledge of the trainee for the given topic. Given this, we found that the available training fails to accomplish these goals and the content does not translate into actionable skills and knowledge to effectively use the Modria system. While descriptive video might be adequate to introduce end users to the basic concepts of ODR and other informational purposes, it is less effective for training interaction with software. A more effective method would be through simulation training using an interactive eLearning module that allows users to experience the software outside of the context of their case. Navigating the user interface of Modria was often confusing and several features did not work as expected. It is often unclear if you have performed a process correctly due to lack of feedback from the software itself, even in a situation in which we were in direct communication via video conference in non-adversarial conditions. Further, we noted that even when both parties are logged in at the same time sending updates, the dispute does not update with new information without refreshing your browser or clicking on the link provided in email notifications of an update. Nothing within the system or training indicates that this is necessary, and the researchers only discovered this through trial and error. This issue, especially given the lack of integrated video/audio communication options, introduces a significant barrier to timely and effective communication between parties engaged in ODR. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 17 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 While the text chat feature in Modria is functional, we found the lack of video and/or audio communication features to be a key concern when considering accessibility within the software. According to a report from the National Center for Education Statistics (Krenzke, T., 2020), approximately 29% of adults in the State of New Mexico operate at or below a Level 1 literacy level, which they define as follows: Adults at this level can be considered at risk for difficulties using or comprehending print material. Adults at the upper end of this level can read short texts, in print or online, and understand the meaning well enough to perform simple tasks, such as filling out a short form, but drawing inferences or combining multiple sources of text may be too difficult. Adults who are below Level 1 may only be able to understand very basic vocabulary or find very specific information on a familiar topic. Some adults below Level 1 may struggle even to do this and may be functionally illiterate. This functionally renders the Modria platform unusable by at least 29% of New Mexicans and is an issue that must be addressed with options that allow users to communicate through integrated video and/or audio. Additionally, Modria lacks other language accessibility features for users whose primary language is in one other than English. Note that the problems identified through the researchers’ experience with using the Modria program also mirror concerns expressed by the participants in the interviews, focus groups, and online survey. They are reflected in three of the coding themes in this study (see next section): ● Barriers to Effective Utilization of Modria: Ease of Use,Language, and Other Accessibility Features ● ODR Should Have a Video Component/Live Mediator Option ● Technical Issues with Modria Interface/Processes ● Difficulty Accessing Modria/Participation 2. In-Depth Qualitative Individual Unstructured Interviews 3. Focus Group Research 4. Online Qualitative Survey Data from the individual interviews, and the focus groups was compiled and analyzed in order to identify common themes expressed by the participants in response to the following open-ended questions: ● What do you consider to be the key determinants of effectiveness for ODR? ● What do you consider to be the current strengths of the AOC ODR program? ● What do you consider to be the current weaknesses of the AOC ODR program? ● What do you consider to be the greatest opportunities for improving the effectiveness of the AOC ODR program? ● What do you consider to be the greatest threats or barriers to the effectiveness of the AOC ODR program? ● What are the most important things that you would recommend be done to improve the quality, and effectiveness of the AOC ODR program? PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 18 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 The conceptual structure underlying the coding of the interviews and focus groups, and the labeling of the data was allowed to emerge and evolve over subsequent interviews (Miles & Huberman, 1984). These coding themes, along with further review of other research, were also used to provide guidance in coding and sorting of data from the online survey, so that the final coding themes were developed which defined the conceptual structure for the emerging understanding of the challenges, and the opportunities for improving the effectiveness and efficiency of the AOC ODR program, and the Modria program in particular. These coding themes are: 1. ODR Potential to Improve Access to Justice in NM in a Cost-Effective Manner 2. The Modria ODR Program has Been Ineffective and Inefficient 3. Barriers to Effective Utilization of Modria: Ease of Use,Language, and Other Accessibility Features 4. ODR Should Have a Video Component/Live Mediator Option 5. Technical Issues with Modria Interface/Processes 6. Difficulty Accessing Modria/Participation 7. Debt and Money Due is an Inappropriate Case Type for Modria 8. Inadequate Training/Communication – Confusion About the Program 9. Issues and Concerns Regarding Legal Processes 10. Time Constraints and Deadlines Impede the Settlement Process 11. Need for Improved Planning and Implementation Strategy 12. Miscellaneous Suggestions for Improvement Each of these themes is discussed in this section, with sample quotes from the interviews and focus groups used to illustrate the meaning behind the themes. Additionally, the survey responses are compiled into thematic categories using content analysis, and are presented unedited in Appendix 1. Note: Comments from interviews, surveys, and focus groups have been reproduced verbatim and have not been edited for content or grammar. 1. ODR Potential to Improve Access to Justice in NM in a Cost-Effective Manner Respondents in the interviews, focus groups, and survey without exception, strongly expressed their hope that the AOC ODR office can learn valuable lessons from the Modria pilot program. Comments supported the desire to more effectively utilize various ODR technologies and methods to improve access to justice for the citizens of New Mexico. ● ODR can be the road to increased access to justice in New Mexico … Look at the big picture, what is the need ODR is intended to address, and at the granular details how will ODR be designed, deployed, evaluated, and managed. ● There is a huge opportunity here. ODR may offer unique and essential benefits for the future of the administration of justice in New Mexico. First-time impressions are a one-time opportunity. We needed to maximize the potential of the pilot project … we need to pause, to reflect on the lessons learned so far, and plan how to go forward to achieve the greatest possible success. ● The New Mexico leadership working on this project needs to be encouraged. New ground has been broken. Vast new potentials have been opened up. Give the good people working on this PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 19 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 project the time, resources, and guidance to go forward. There is a rich opportunity to be harvested. ● The project must be leveraged, not as a failure but, as a successful first foray into innovation. There are lessons to be learned. There is the discovery of the potential for how online services will enhance the administration of justice. ● With the right approach, the right types of cases, and with the support of court clerks, judges, consumer advocates, attorneys, the broader ADR community, the law school, and successful disputants, ODR can be a real boost to the cause of providing better access to justice for the people of New Mexico. ● The possibilities of a strong online interpersonal mediation program are almost limitless for many types of cases. People are used to online communication now … we should take advantage of the opportunity this presents to us. ● During this pandemic, people have become more accustomed to online technologies like Zoom. There is an opportunity here … ODR can work to provide access to justice in an affordable way. ● We should take advantage of technology and redirect resources to improve how technologies like Zoom can make online dispute resolution really work in New Mexico. ● By making good use of available technologies to deliver mediation and other dispute resolution services, we can begin to overcome problems of geography and poverty in a cost-effective way. ● We have to face it. New Mexico is a poor state, and we can’t afford a judicial system that can serve all the people. To bring meaningful justice, we have to leverage our resources – and ODR can be the way to make that happen ● ODR offers the Judiciary an opportunity to further its goal to increase access to justice throughout New Mexico across economic, social, and geographic lines. This program didn’t work, but learn from it and grow ODR in things like traffic cases and compliance cases. ● Modria is a bad fit for the project. The platform was misrepresented as to purpose and capability, has serious design flaws, and works poorly. However, learn from this failure and develop new ODR programs and services that can increase access to justice for the people of New Mexico. ● The way to improve and grow ODR is through data … we need to show that it saves time and money. This program didn’t do that, but there are lessons here that we can use to improve how we use online mediation and other methods to resolve disputes. 2. The Modria ODR Program has Been Ineffective and Inefficient Respondent comments reflect a broad dissatisfaction with the effectiveness and low cost-efficiency of the ODR program for debt and money due cases. Between the pilot program dates of June of 2019 through July of 2021 the agreement rate among debt and money due cases was very low, with only 2.42% of all cases referred to ODR resulting in an agreement: TIMEFRAME Total Cases Referred to ODR Total Agreements Made Percentage of Agreements Made PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 20 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 2019 (June-Dec) 864 15 1.73% 2020 (Jan-Dec) 640 34 5.31% 2021 (Jan-July) 5267 115 2.18% TOTALS 6770 164 2.42% 3. (See Appendix 2: AOC ODR Reports December 14, 2020 – June 30, 2021) We were told that the estimated budget for the Modria ODR program over the two-year pilot program was approximately $460,000 (Note: the researchers could not verify this amount). Even if the actual cost of the pilot was only half of that, the Modria pilot program has clearly not been cost-effective. ● Judges want cases closed. Legislators need to see money saved. The Modria program did neither. ● The most effective dterminate of success is participation followed by settlement rate. With the rise of one hopefully leading to the rise of the other … it was not used much by either parth. The greatest tool in the world is not helpful if it isn’t used. ● I think this is a creative idea, but it needs some work. Our cases that were going to ODR normally go to our Arbitration program, which has a successful participation rate. Most of the cases that went to ODR did not settle/were not resolved, so it just took longer for them to get to our arbitration program, where the case was then taken care of. ● ODR has not proven to be cost effective. It is not making a significant difference in the debt/money due caseloads and has not proven to have value for either New Mexican consumer or AOC. During the pilot in the summer/fall of 2019, less than a dozen cases reached a negotiated agreement through the ODR platform ● We represented a creditor (public school district) owed money by a former employee. An attorney had entered an appearance but failed to answer. We filed for default, and that was when we received an invitation to the ODR program. At that point, we saw no benefit to ODR, as we were almost certain to be granted the default. We might have participated if we had been shown how the program could have been of benefit. 3. Barriers to Effective Utilization of Modria: Ease of Use,Language, and Other Accessibility Features Unsurprisingly, the lack of language accessibility features within Modria/ODR is a pain point that must be resolved for any program to be deemed successful. In trying out the pilot case set up for them, we found the Modria system to be frustrating overall, and in particular, the asynchronous nature of the platform (particularly with its reliance on email communications) to be cumbersome and difficult to use. (See findings of the Practice Case for Researchers to Use Modria discussed earlier in this section.) ● If ODR is to work for any type of case, we have to have universal access. The diversity of the people of New Mexico demand that. ● My grandma is a smart person, but she just refused to try it. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 21 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● It’s just too impersonal for a lot of folks. I think people just want to be heard. You know, to have their day in court. ● The Modria system is not user-friendly, and is not intuitive to use. ● Very confusing to use – reliance on email as a part of the process was not clearly explained, and caused unneeded delays in the process. ● Tyler doesn’t work with deaf or blind users (sic) without having screen readers. ● It’s hard to manage quality and good communication with a text-only system. ● ODR is not available to Spanish speakers. ● Add multiple languages especially Spanish and Navajo ● Tyler doesn’t work with all smart phones. ● Did not provide ADA accessibility, in spite of claims by Modria 4. ODR Should Have a Video Component/Live Mediator Option The comments almost uniformly reflected a strong feeling that ODR (at least for this type of case) cannot be successful without in person/video conference mediation. Asynchronous chat does not allow proper dispute resolution to take place. We believe that this is a crucial feature for any future ODR program for this case type. People want to feel that they have been heard, and “seen.” (See findings of the Practice Case for Researchers to Use Modria discussed earlier in this section.) ● I really think you need to bring in more interctive video to make ODR work. Text-based technology can be part of it – like a hybrid system – to do ancillary work in the ADR process. ● Any text-based ODR program needs to have a place in the platrorm where parties can request a mediator or facilitator. ● The program needs to have a video module fully integrated into the system to stay within Odyssey. ● Video training should be available in several languages, and within the ODR modules themselves. ● For mediators, settlement facilitators … all those people who are experienced in ADR, there was … there is … a heavy learning curve to use text for mediation and facilitation. It is too linear for all but the most simple, transactional communication. Resolving disputes requires more interaction between the parties and the mediator. ● Add a portal in the platform to allow a meeting in a video conference to discuss matter. ● Have a mandatory meeting via teleconference or phone to discuss matter as part of process. ● Have mediator available to help ● Integrate zoom mediation with live mediators in lieu of asynchronous chat ● Give the portal the ability to schedule a videoconference PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 22 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Need to actually have contact with the other side ● Highlight mediation as an option (cases that used mediators seemed to settle more often than those that did not). Make sure mediators have the ability to work within the program to submit documents, etc. ● Allow for telephonic participation in the program, for those parties without email or internet access. 5. Technical Issues with Modria Interface/Processes Based on the comments from respondents, and from the experience of the researchers, Modria is a poorly designed program with an inflexible (and often nonfunctional) interface and features. The researchers found the system to be opaque and confusing. When finding a replacement, AOC needs to develop a list of core features that must be present in order to move forward when evaluating possible replacements for Modria in the future. I would recommend development of an RFP to bring transparency and accountability to the process. Furthermore, several comments claim that features that do in fact exist are not present in Modria, which speaks to the strong need for the development of an effective, professionally designed training program. ● I believe that Tyler misrepresented the program – the software was compromised. Judge Levy is a hero. There were many more problems in the beginning, but she bludgeoned Tyler into making needed improvements. Even with that, the platform doesn’t work well. ● Tyler-Modria is frustrating for me to talk about. It was over-sold. The reality is that it’s clunky, not user-friendly, and not designed with the user in mind. ● Tyler sold this program as a platform that would work for all cases. It became money due because it didn’t work for other cases – Tyler mis-stated. They say they want to work with us to fix the issues with the program? I don’t trust them … they just want to bill the AOC for more money. ● This program (Modria) will never meet the needs of New Mexico, especially for debt and money due cases. ● I am a millennial and my overall experience with the platform is it was difficult to use, there were so many steps to log in to a single matter. There were tedious steps to enter an initial offer. The biggest downfall was getting the other side to participate, either because they were unaware of the program, or they did not understand how the platform works. ● The user experience is disjointed and frustrating. ● There were no formal means to re-refer a case to ODR. There were times when we had to reach out to Tyler directly to re-refer, because Plaintiffs were opting out of using the program. There was no tracking of these cases. ● I saw times where a money due case was on the brink of a settlement … in the midst of a settlement, and the software timed out the case. There is an automatic 30-day timeframe limit in the software. That’s nuts. The software needs to serve justice, not itself. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 23 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● The program did not always allow us to have private communications with the mediator. That is a bug that needs to be worked out. ● Allowing more flexibility in payments. Not requiring all payments to be equal. ● Allow for a monthly payment plan that extends over a certain number of months which specifies the court will keep the case on an inactive calendar (but not subject to dismissal for LOP) until the time for repayment ends. ● Make it easier to upload documents so in cases where there is a dispute everyone can look at the document just like they were doing an in-person mediation. This took a lot of time to resolve with tech support. ● Post the answers to the complaints in ODR so the mediator can see all the pleadings to save time and frustration for the litigants. ● Received information that the attorneys were not able to add their paralegals to the system, and that firms were unable to add the email of the attorney working on that case. ● Problems with the ODR platform. We received many calls saying Tyler was not helping them and asking what they should do. If there is a way to have a Tyler team assigned for ODR platform questions, that would be helpful. I think having one person to troubleshoot ODR is very difficult. ● The process was long and added additional legal fees for Plaintiffs with little to no resolution. Make it easier for the Defendants to participate and more resolution will occur in my opinion. ● On the written notices provided, it was not clear who the parties were requiring us to look up the case numbers, etc. Those should include the caption. ● Each attorney receives 100's of emails per day and to find one from this ODR program to respond to is unrealistic. The program needs to be ran with a real person facilitating the agreements. ● We should be able to to offer a discounted interest rate if the defendant is willing to make payments (See findings of the Practice Case for Researchers to Use Modria discussed earlier in this section.) 6. Difficulty Accessing Modria/Participation Many of the respondent comments bring up known issues with internet accessibility through cell phones, kiosks, etc for areas with poor internet access. Further, there is clearly an issue with getting actual participation after referral from both plaintiffs and defendants. While there are obviously going to be issues with participation from defendants who are avoiding any communication regarding their debt, comments indicate that plaintiffs also usurp the process of ODR by refusing to engage and then filing motions upon completion. We’ll touch on this more later, but oversight of the process to help ensure participants engage in good faith is potentially a key consideration for any future programs. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 24 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● We need better access for ODR in many, if not most, areas of the State. There are problems with basic computer skills, and even getting to a computer. ● Maybe we could use remote pro bono legal clinics – we might be able to set up a program that way. ● Allow pleadings to be accepted even though ODR has not been completed. We are getting more and more stating they do not have internet access. ● ODR should not be referred until the defendant has responded. I referred many many cases after service had been effected, and ODR ended unsuccessfully due to a lack of participation. Most of those cases ended in default judgments. ● Making the program mobile friendly (like an app) would be helpful. We have lots of pro se customers who don't have computers but have smart phones. Also having kiosks in the computer that allow you to access your ODR portal (for those who don't have computers or cell phones) ● Kisoks are growing as one-stop portals in courts. ● Regarding kiosks in Courts – safety concerns are always there – but there are Sheriff’s deputies at courts – some of the smaller courts probably don’t have security … cyber security issue ● In providing the pro se defendant's phone number or other contact information, we should be able to indicate that this is our last known information ● Remove the Plaintiff's ability to end ODR, since the Supreme Court order indicates plaintiff participation is mandatory ● You should clearly state whether the plaintiff's voluntary election not to participate is without some form of penalty that might be imposed by the Court. ● I think you are on the right track. However, I live in the southeast part of the state and there are not many resources available for people dealing with the courts. When programs do become available, they often don't know about them. Many of the customers tend to be leery of what is being offered because they are nervous when dealing with the courts. Thank you. ● There needs to be an incentive for plaintiffs to participate. Many plaintiffs just opted out or waited until the 30 days of ODR were over, and submitted their motions. 7. Debt and Money Due is an Inappropriate Case Type for Modria There is a strong feeling that traffic cases are the appropriate case type to pilot ODR. This feeling falls in line with the researchers finding that ODR for debt and money due should be put on hold while the traffic program moves forward. Lessons learned from that program along with strategic planning efforts could yield positive results for other case types being added to the program moving forward. ● ODR could be considered for traffic ticket cases. ● ODR has an adverse impact on the due process rights of defendants in debt and money due cases. There is an imbalance of power in these cases and using ODR exacerbates this imbalance. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 25 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● There is well documented abuse in the debt collection industry - see FTC reports and CFPB reports. ODR does not provide judicial review of these cases to guard against these problems. ● ODR in debt/money due cases has a lot of potential for abuse, and its use to date has not proven to be cost effective. Recommend the AOC discontinue the use of ODR in debt/money due cases ● Great idea and commendable efforts to try to make this program a success. Unfortunately, debt/money type cases are not optimal for online dispute resolution and no matter what changes are made, the cost will continue to outweigh its effectiveness. ● Debt and money due cases are the wrong thing for this platform. We were trying to get information to defendants involved in the process who have a history of avoidance. If ODR isn’t relevant to them, if they don’t see that they can benefit from it, they won’t pay attention. ● This platform is not a good fit for debt/money due cases. It might be a better fit for traffic cases. ● National research on ODR and experiences in other states raises similar concerns from consumer attorneys about problems with ODR in debt cases ● AOC could improve the program by doing a pilot in traffic ticket cases only. 8. Inadequate Training/Communication – Confusion About the Program Well-designed training for end users, as well as training for court employees who would expect to answer phone calls regarding questions from end users is essential for future success. While technically there are instructional videos available, the results of this survey do not indicate that users are aware of their existence. In their own experience role-playing the parties in a simulated case, the researchers found the videos to suffer from poor instructional design and generally were inadequate for properly preparing users to engage in ODR through Modria. Access (although limited in actual utilization) to a recorded video alone is an inadequate training tool for a program of this type. An eLearning program would need to be designed to guide users through a mock session to fully understand functionality and processes. Furthermore, there is a strong feeling that respondents placed in ODR have little to no guidance on how they should reasonably proceed. This issue is separate from being trained to operate a piece of software but built-in resources for respondents could be an effective feature for assuaging the fears of consumer advocacy groups that strongly oppose ODR. ● It’s just too impersonal for a lot of folks. I think people just want to be heard. You know, to have their day in court. ● The problem is, the parties just don’t know what’s going on with the asynchronous system. ● Tyler’s training didn’t really address how to actually operate the program. ● You couldn’t even call it training – it was more like a promotional presentation. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 26 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● I believe having training in general civil will give us more knowledge on answering calls. Most callers have other questions in regards to their case and we don't know how to answer/assist them. ● Call the opposing party after one of the first party has created the initial offer. ● More explanation to case participants. They don’t understand the process. ● Better communication with the public when they are referred to ODR. ● There was confusion about what to do once the case received a non-participation. Should the case be re-referred to ODR? Should the case go onto the judges docket? Should the case go to our arbitration program? We had many plaintiffs who were just opting out of the system immediately. It didn't seem like there was any penalty for them, and therefore there was no incentive to actually try to use the ODR program. It also confused Defendants who wanted to use ODR when the plaintiffs just opted out immediately. ● The distribution of information about the program. A flyer or information sheet should be included with the Complaint and/or summons to be served on the defendant, and included with the form for the Answer to the Complaint. ● The defendants don't really understand the process. ● More training w/ Modria's portal. Maybe having more access to update information and etc. ● Provide examples of what the program looks like when parties participate (e.g. screenshots, videos, etc.). ● ODR court personnel do not share the knowledge to other court staff regarding ODR program (lack of communication) ● A lot of the people have emails addresses and ok with the facilitations happening telephonically, but they need a real person to talk to when they have issues or questions. The State needs to show good customer service if they are going to require these cases be referred to this program. The attorneys are not very responsive to the email system. (See findings of the Practice Case for Researchers to Use Modria discussed earlier in this section.) 9. Issues and Concerns Regarding Legal Processes There are a lot of comments asking to be able to use a stipulated judgment during the process, and we are unsure of the legal issues surrounding this. Regardless, this speaks to a lack of understanding of how ODR is positioned within the technical legal process. FAQs explaining what is legally appropriate and WHY should be considered for inclusion in a future program. Additionally, this thematic area brings up several comments (likely from respondents in consumer advocacy groups) regarding oversight to ensure there is no abuse of the legal process from the plaintiffs. The researchers feel this is an interesting point that should be considered for future programs. There is an obvious manpower issue regarding the ability to achieve this, but even the ability to request review if a participant feels like the process is being abused would both assuage fears from PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 27 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 the groups making these comments, but likely also have a chilling effect on abuse occurring in the first place. ● The mediator should be able to upload Judgments. My mediations have often been resolved via Judgment and the platform does not permit the mediator to upload it after the parties have reviewed and agreed. This makes no sense. ● Also, we received feedback from some attorneys that when they chose to use the mediation feature, they were not able to speak with the mediator one-on-one in the chat function or exchange documents with the mediator only. They said they were able to upload, but that the documents were there for the other party to see as well. ● We are not okay with agreements that are not signed off on by the judge. ODR does not allow that. ● Do not report the Plaintiff as failing to participate if exercise their option to opt out of settlement negotiations. Either don’t provide the option to withdraw or dont report as failure to participate. ● Cases should only require ODR if answer has been filed. ● Allow for a Stipulated Judgment option, monthly payments but also attorney fees and court costs. Explain the stipulated judgment will be filed with the court and end the lawsuit provided payments are made. ● As an attorney for the creditor, we have very strict settlement guidelines, and we use those figures when putting in what would be possible. Having a Defendant propose a different amount is rarely approved without a documented severe and permanent hardship, and the back and forth is futile. ● Determine if the case is ripe for mediation. In my only interaction with the program the defendant had not answered, and a default was in process but there was no method to stop the mediation process. ● ODR should provide a consent judgment resolution option. ● Most Defendants have a Debt Management Company they are making payments to and that they have given a Power of Attorney to. The DMC rarely hires counsel and is not part of the Modria discussion so the case really can't be settled as they are holding the settlement funds. 10. Time Constraints and Deadlines Impede the Settlement Process There are a variety of issues regarding deadlines and time constrictions. Overall, our recommendation would be to add flexibility to the system (barring any legal reasons why this is not possible). ● On occasion, I have needed more time to resolve a case. I am unable to ask for more time directly through the platform and must go through another person. The direct route makes more sense. For example, it took days for my administrator to get to my request for more time. In the meantime, the case expired and dropped out of my dashboard. I was very close to resolution, and it disappeared. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 28 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● In ODR, there was a 15-day deadline for the Defendant to file an answer once the case finished in ODR. This ended up being more strict/more difficult for Defendants than our normal arbitration process because with arbitration, as long as the Defendant filed an answer before the plaintiff filed for default, the case would go to arbitration. Sometimes the plaintiffs don't file for default for months, so this gives the Defendant a longer period of time to file an answer. ● Give the mediators more time after sending the email they have been appointed. For those of us who still work, I do not check my work email in evenings or on weekends and I miss a deadline to post a welcome message. Thus, being appointed on a Friday with a deadline of Monday or Tuesday is not practical. ● Mediator should have the ability to extend the time limit of the mediation if needed 11. Need for Improved Planning and Implementation Strategy These comments reflect both the frustration with the lack of proper implementation planning for the Modria pilot program, and the desire to engage in a highly participative and inclusive planning process to help ODR move forward in New Mexico, to provide better access to justice for all New Mexicans. Strategic and operational planning aims to guide an organization as it moves forward by establishing a clear mission and vision for a program, and by developing goals and actions that align with that mission and vision. The common perception is that the ODR program did not follow that approach for the Modria pilot project implementation. It should also be noted that projects such as ODR are used to carry out strategic goals, and depend on a project manager to see the project through to completion. It appears that while the ODR project had very good and dedicated staff to support the ODR pilot project, there was no central project manager dedicated to managing the planning and implementation process. From our perspective, the pilot project’s success appears to have suffered from not having a strong project manager to guide the implementation process for the ODR project. As such, proper strategic and operational planning, implementation, on-going program evaluation, and re-planning for continuous improvement of the ODR project did not occur. The importance of using a participative, high-involvement strategic and operational project planning process cannot be overstated. Successful planning models depend on the emphasis and organization of the particular planning process and steps (Weimer, D. and Jonas, P. 1997). Moreover, full participation and communication by diverse stakeholders in the legal community from the very beginning of a project will serve as a uniting force within the process. When people participate in a project planning process, they are more likely to be committed to the plans, because they have talked them through, differences have been worked out collaboratively so that the project plan reflects their own thinking, and because the group has developed consensus. (Kanter & Spencer, L. 1989) Respondents commented in support of the need to follow a participative project planning methodology, including a broad range of key stakeholders including both supporters, and those who may have concerns about the use of technology in the administration of justice. Collaborative planning can develop synergy from diversity, resulting in a stronger project, and a more successful outcome. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 29 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● A strength of the project has been that there are good people working hard to make the pilot project a success. JID, AOC, and court staff, as well as volunteer mediators, are competent, conscientious, dedicated and making extraordinary extra efforts to support the pilot project … the concern here is to strengthen their dedicated work with a more organized infrastructure. ● The New Mexico leadership working on this project needs to be encouraged. New ground has been broken. Vast new potentials have been opened up. Give the good people working on this project the time, resources, and guidance to go forward. There is a rich opportunity to be harvested. ● This program needs champions. New Mexico has long been a leader in ADR across the country, but now there seems to be a leadership gap for ADR, which has affected ODR’s ability to grow and thrive. Who are potential champions for the program> Who are the natural allies for ADR and ODR? We need to find them and get them involved. ● Put together an ODR task force to examine ODR’s areas where we could improve standards and best practices – what’s working and what isn’t – not only for the Modria project, but for the future development of ODR technologies in the State. ● There needs to be a role for the Law School in all this. I’m not sure what that is, but they need to be involved and be champions for ODR in the state. ● The project must be leveraged, not as a failure but, as a successful first foray into innovation. There are lessons to be learned. There is the discovery of the potential for how online services will enhance the administration of justice. ● Develop a strategic plan for ODR in New Mexico. Look at the big picture, what is the need ODR is intended to address, and at the granular details, how will ODR be designed, deployed, evaluated, and managed. ● Slow down and test. Remove any deadline for a rollout. Systematically, thoroughly, and repeatedly test the program with a variety of types of users. This will have major benefits including to have a reliable, fully functioning ODR platform; and to build widespread buy-in two making the program work, by having various stakeholders and users participate in the testing. ● The project appears to have been created to install Modria, rather than by creating a needs analysis, a vision, a mission statement, a strategic plan, a request for proposals, not starting on a small, manageable scale with sufficient time for development through experience, and with a rush to deploy – it was “look after you jump.” ● There wasn’t enough of a high-involvement planning process when the program was implemented. We don’t really have any strong champions … in fact, we have people who were not really involved in planning ODR who are now … well, detractors and even enemies. That’s no way to start a new program. ● AOC could improve the program by listening to consumer advocates who have raised concerns about the use of ODR in NM and specifically in debt and money due cases. ● Listen to the input of consumer advocates and experts on ADR-ODR to improve ODR. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 30 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● The imperative was to implement the program, not to take the time need to properly plan and implement something of this scope. We were building the plan as it was flying. We rolled out what should have been a 2-year implementation project, in a very short time. It shows. ● There was a continual rush to meet a new rollout date … the crisis mode of operation compromised performing methodical, planned, and organized work. ● The focus on the product (Tyler-Modria) did not afford New Mexico with the opportunity to develop a strategic plan for the design, deployment, evaluation, and management of a major new ADR program, including how to enlist the support of, and educate and prepare every person and organization which might be impacted by the program. 12. Miscellaneous Suggestions for Improvement In addition to those suggestions referenced above, respondents had a variety of recommendations for improvement of the overall ODR program, as well as for the Tyler-Modria platform. One recommendation of note is that any new ODR program follow a “continuous improvement” model of implementation with on-going user feedback, program evaluation, and making adjustments and changes to improve the effectiveness of the program. The University of Missouri School of Law published an excellent research paper “emphasizing user-centric designs for Court ODR” (Kulp, H. and Schmitz, A., 2020). Supporting this approach. ● Slow down and test. Remove any deadline for a rollout. Systematically, thoroughly, and repeatedly test the program with a variety of types of users. This will have major benefits including to have a reliable, fully functioning ODR platform; and to build widespread buy-in two making the program work, by having various stakeholders and users participate in the testing. ● It’s really important that any ODR program be designed from the user’s perspective. We need to do that moving forward … make sure that whatever we do is field-tested several times before the program is rolled out for real. Then, do continuous assessment of the program with feedback from the users, and make improvements along the way based on that input. ● Build in a continuous improvement process into the program. That should be done for any new ODR program we implement. ● Convene a regular forum of users to share lessons learned, and best practices in using ODR – what works well? What doesn’t work? Building a community of users to constantly learn more about the ODR program and process, so it can keep getting better. ● There needs to be a role for the Law School in all this. I’m not sure what that is, but they need to be involved and be champions for ODR in the state. ● Reimagine the flow (of ODR) to make it more cohesive for the parties. Tracking each party referred to ODR to completion, to even what happens in their case. ● Monitor each case referred to ODR and add more human interaction not move the case along when needed. ● Video mediation with trained mediators; for district court, perhaps retired attorneys could be used? PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 31 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● A dedicated, trained staff person to facilitate ODR. This should be that person’s only job. ● A dedicated, trained call center to be a one-stop location for answers from technical troubleshooting, to answering questions about what ODR is and how to use it. ● Training for judges as to what it (ODR) is and how to enforce its use. ● Computers in court houses and other locations by parties who do not have connectivity/phones/computers for ODR. ● More buy-in with justice partners, such as the consumer advocates, and attorneys who file these cases. ● We only have a limited pool of attorneys for the program, maybe a rule change re: what qualifies as lo bono or pro bono. ● A pool of funds to pay a small fee to do mediation. ● More training for paralegals, retired, etc. to grow the pool – could probably handle money due cases. ● For Metro and Magistrate courts – plenty of non-attorney mediators who are (could be used) ● (Any software used in ODR) has to talk with Odyssey. ● (In response to question, “Where would you look to be moving next?”) – o Evictions program – magistrate and municipals – after the funding goes away o Money due (still has barriers) o Foreclosure? o Administrative child support (Coordinate CSEB) o Where there is a heavy self-represented litigant pool. ● Text makes sense for a hybrid model of ODR. It doesn’t seem to really work as the only communication in money due cases. ● This program needs champions. New Mexico has long been a leader in ADR across the country, but now there seems to be a leadership gap for ADR, which has affected ODR’s ability to grow and thrive. Who are potential champions for the program> Who are the natural allies for ADR and ODR? We need to find them and get them involved. ● Network with others in the field to build continuous improvement teams who will in turn help to promote ODR in the legal community, and in the public. ● On-going continuous improvement should be embedded in the program administration – on￾going evaluation and assessment to keep making the program better and more effective. ● Learning is in the trenches … continuous learning … continuous collective feedback among the mediators. ● Can a triage process from staff work with the system, so that cases can be placed with the most appropriate ODR process? PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 32 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● There needs to be a lot of effort put into marketing the program, so that people are aware of it. Flyers alone just don’t work to get the word out to everyone who should hear about ODR. ● How much can we expect from volunteer mediators? Do we need incentives? What about MCLE credit for attorney volunteers in the ODR program? 5. Secondary Research and Document Review A review of relevant literature and other research provided the researchers with cognitive models, best practices, and recommendations concerning ODR. That information has been incorporated (with appropriate references) into the Recommendations section of this report below. Additionally, several of these research articles and reports are attached in Appendix 8 of this report. A number of these research reports and articles were used for reference and comparative analysis purposes. The documents which were collected and reviewed included research reports, conference proceedings, studies, legislation and public policy recommendations, along with other information pertaining to the adoption and implementation of online dispute resolution (ODR) programs and technologies. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 33 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 RECOMMENDATIONS TO AOC ODR OFFICE In this section of the report, the researchers present their recommendations for the AOC ODR program. Each recommendation indicates the thematic areas for improvements that they address. The researchers’ recommendations being made are listed below, with the coded themes listed that these recommendations address. Note too, that the previous section of this report (Research Findings and Supporting Data) includes many respondent suggestions and recommendations for improving the program. The coded themes referenced at the beginning of each recommendation should be referred to for respondent comments and suggestions, and further discussion by the PRAXIS research team. RECOMMENDATION 1: Discontinue the Current Tyler-Modria ODR Program This Recommendation Supports the Following Themes: 2. The Modria ODR Program has Been Ineffective and Inefficient 3. Barriers to Effective Utilization of Modria: Ease of Use, Language, and Other Accessibility Features 4. ODR Should Have a Video Component/Live Mediator Option 5. Technical Issues with Modria Interface/Processes 6. Difficulty Accessing Modria/Participation 7. Debt and Money Due is an Inappropriate Case Type for Modria 8. Inadequate Training/Communication – Confusion About the Program 9. Issues and Concerns Regarding Legal Processes 10. Time Constraints and Deadlines Impede the Settlement Process See commentary for the above themes in the Research Findings and Supporting Data section. First and foremost, the researchers recommend the discontinuation of the current ODR program for Debt and Money Due cases on the Tyler-Modria platform. It is clear from information gathered through interviews, surveys, and the researchers’ own experience that Modria neither functions as was promised by Tyler Technologies, nor in a fashion conducive to a successful ODR program for this case type. AOC should be wary of the sunk cost fallacy, as continuing the current program could have a further damaging effect on the wide scale adoption and success of ODR programs for other case types or a potential relaunch of ODR for debt and money due cases. RECOMMENDATION 2: Capitalize on Opportunities for Broader Use of Appropriate ODR Programs, to Improve Access to Justice for People of New Mexico This Recommendation Supports the Following Themes: 1. ODR Potential to Improve Access to Justice in NM in a Cost-Effective Manner 11. Need for Improved Planning and Implementation Strategy 12. Miscellaneous Suggestions for Improvement See commentary for the above themes in the Research Findings and Supporting Data section. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 34 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ODR is the future, the future is now, and now is the time to boldly move forward to design and implement successful ODR programs in New Mexico, to improve access to justice for all of our citizens. In his article, Online Dispute Resolution: The Future is Now, Brent Batten (2020) states, “As our population base ages, individuals who are adverse to using technology are becoming a minority, and the demographics of mediation clients is changing. Mediators are now dealing with generations that are extremely adept and flexible in the technological fields and are more comfortable with technology than ever before ... … Technology has shaped industries for years, and adapting is critical. Our field is no different. We must adapt and respond to the evolving needs of participants. We must provide people with options that make us accessible and increase their comfort. Only in this way will we continue to be leaders in conflict resolution.” The application of ODR to dispute resolution (ADR in particular), can transform court procedures and dispute resolution in a revolutionary way. With proper design, implementation, and continuous assessment and improvement, ODR will increase access-to-justice, especially for self-represented litigants, while making court operations more efficient and cost effective. (National Center for State Courts, n.d.). RECOMMENDATION 3: Convene a Diverse Task Force of ADR, ODR, and Stakeholder Experts to Review Research and Best Practices in the Design and Implementation of ODR Programs, in order to Establish ODR Program Standards for New Mexico This Recommendation Supports the Following Themes: 11. Need for Improved Planning and Implementation Strategy 12. Miscellaneous Suggestions for Improvement See commentary for the above themes in the Research Findings and Supporting Data section. Over the past several years, state and local courts have launched dozens of ODR pilot projects with varying degrees of success, and the lessons from those experiences can be valuable developing successful ODR programs in New Mexico (National Center for State Courts, n.d.). Several articles describing best practices in the planning, design, implementation, and continuous assessment and improvement have been included in this report in Appendix 8. The National Center for State Courts (n.d.) assessed several pilot ODR programs, and has developed eight lessons for ODR implementation that have been derived from the experience of working with those pilot programs. In summary, those “best practices” include: 1. Before you begin, understand what ODR is and what it isn’t. “While the design and functionality of an ODR platform is important, courts must also recognize that simply plugging in a well-designed platform is not sufficient in and of itself to assure a successful pilot program ... the platform must fit into a functional infrastructure” of the legal process to protect disputants’ due process rights and to support informed decision-making by the parties. The NCSC report also states that the launch of successful ODR programs has, in large part, been predicated on taking time to assess the PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 35 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 existing legal and operational infrastructure and investing resources to ensure adequate support for ODR.” 2. ODR projects can benefit from effective judicial and administrative leadership. This also was an important issue raised by respondents in this program evaluation study. The NCSC found clearly that in planning, design, and implementation of successful ODR projects, the importance of having judicial and administrative champions for the project cannot be overstated. A consideration in identifying such champions is that the ODR planning process can take time, so that the continuity of leadership is important part of identifying champions. Of note, several respondents in this study noted that there is an absence of ADR (and by inference, ODR) leadership in New Mexico. Such leadership and identification of ODR champions must be cultivated. 3. Identify and articulate the business problem(s) you want ODR to address. In planning for the implementation of ODR projects, NCSC found that ODR programs first articulating the business problem and outcomes that they were trying to address with ODR (e.g. reducing default rates, increasing litigant participation, expanding access to justice, etc.) were more successful. It is essential that understanding those business problems/outcomes be integral to their planning. One example of this For example, one jurisdiction was trying ODR because they wanted to expand access to justice for self-represented litigants, and prioritized easy-to-use platforms that provided plain language and useful legal information within the ODR experience at low or no cost to the court user. (NCSC, n.d.) 4. Fully integrate ODR with case management and other court technology platforms. A major potential benefit of ODR is the extent to which it can streamline case process both for parties and for the court. Commercial ODR platforms however, are designed to comply primarily with industry standards for integration, rather than court standards. In the report, NCSC (n.d.) noted that: “… integration was impractical when courts operated on legacy systems that themselves were out of compliance with contemporary industry standards. Integration with CMS ensures that the court can monitor case progression and enter court orders and judgments without duplication of effort by either the court or the litigant. Integration with e-filing systems allows litigants to initiate and respond to case filings seamlessly. These platforms could also provide options for payment of court fees, requests for fee waivers, service of process, and other ancillary litigation tasks.” 5. Engage with stakeholders early and often. (See Recommendation 4) The NCSC observed that the importance of stakeholder engagement too often is narrowly as a post hoc marketing strategy. It is of critical importance to engage all stakeholders in platform design, content development, user testing, on-going program evaluation, and continuous improvement, so that user’s needs are better met. Engaging stakeholders also gains support and champions for ODR, which helps to leverage existing resources, and to secure buy-in from stakeholders who have invested in the program’s success. It is important to note too, that internal stakeholders are at least as consequential as external stakeholders. 6. Remain faithful to core principles, but flexible about how court rules and business practices embody those principles. It was found that ODR implementation often got bogged down in PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 36 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 customizing a commercial ODR platform to comply with existing rules or business practices. This led to conflicts with those court rules and business practices that were intended to be operationalized. It is important that ODR serve to support the legal processes of the court, and not the other way around. ODR procedures should further the principles of the court, and if those principles cannot be appropriately adjusted to ODR, then the ODR procedures must be modified to meet those court principles. 7. Process simplification should focus on the user’s experience, not on internal court procedures. The ODR process and procedures must work for the users in order to be successful. Ensuring that the user’s experience with ODR is both helpful and seamless may involve adding more steps to the court’s normal internal processes, but if that improves the user experience, the program is likely to be more successful. While ODR may thus add steps to the overall process, remember that for the users, their overall experience with the judicial process may have streamlined matters from their perspective. 8. Be clear with commercial vendors about project goals and expectations. If using a commercial vendor ODR product, it is critical that the court have a clear idea about project goals and expectations before purchasing a software platform. In evaluating vendor products for use in ODR, knowing very clearly what capabilities that you expect the platform to perform must drive the evaluation of the product. This does not appear to be the process that was employed when the commitment to use Tyler-Modria platform for the ODR debt and money due program. As the New Mexico Courts commit to moving forward to implement well-designed and executed ODR programs, it is important to learn from the successes and mistakes not only from the Modria-ODR program, but from the newly implemented Traffic ODR program, and from the best practices from other jurisdictions. Taking the time to invest the time and attention to lessons learned is necessary for a successful ODR program in the future. NOTE: FOR AOC-ODR TRAFFIC PROGRAM – implement continuous assessment and improvement process as soon as possible, following the principles from NCSC (n.d.) above. RECOMMENDATION 4: Engage in a High-Involvement Participatory Strategic and Implementation Planning Process Focused on the Future Development and Success of ODR in New Mexico. This Recommendation Supports the Following Themes: 13. Need for Improved Planning and Implementation Strategy 14. Miscellaneous Suggestions for Improvement See commentary for the above themes in the Research Findings and Supporting Data section. We are recommending that AOC and the ODR Office engage in a highly participative and inclusive planning process to help ODR move forward in New Mexico, to provide better access to justice for all New Mexicans. Strategic and operational planning aims to guide an organization as it moves forward by establishing a clear mission and vision for a program, and by developing goals and actions that align with that mission and vision. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 37 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 It is also important that this implementation planning utilize a skilled project manager to see the project through to completion, and to guide the implementation process for future ODR projects and programs. The importance of using a participative, high-involvement strategic implementation and operational project planning process cannot be overstated, if the AOC ODR Office is to move forward with lessons learned from this ODR pilot project, to grow a successful ODR Program (See Recommendation 2). Successful planning models depend on the emphasis and organization of the particular planning process and steps (Weimer, D. and Jonas, P. 1997). Moreover, full participation and communication by diverse stakeholders in the legal community from the very beginning of a project will serve as a uniting force within the process. When people participate in a project planning process, they are more likely to be committed to the plans, because they have talked them through, differences have been worked out collaboratively so that the project plan reflects their own thinking, and because the group has developed consensus. (Kanter & Spencer, L. 1989). While developing a strategy is one of the first steps to implementing organizational change, the implementation itself is vital to a project’s success. Without an efficient implementation process, even the best-laid plans may not come to fruition. This appears to have been a key shortcoming of the Tyler￾Modria ODR pilot project. Strategic and implementation is a process that begins with self-assessment and realization and then reorganization to facilitate the success and effectiveness of organizations, programs, and projects. Well-formulated strategies form the basis of success, and should therefore be well developed by the right people (including external stakeholders) and the right organizational levels. Strategy formulation and implementation should be linked by an on-going evaluation and improvement strategy to realize strategy success otherwise the strategies remain useless paperwork. Miller (2020) writes for Harvard Business School that, “Strategy implementation is the process of turning plans into action to reach a desired outcome … success rests on [the organization’s] capacity to implement decisions and execute key processes efficiently, effectively, and consistently.” New project initiatives must be constantly monitored, evaluated, and adjusted if they are to achieve success. He goes on to identify 7 key steps in the implementation process: 1. Set clear goals and Define Key Variables. The old saying, “if you don’t know where you’re going, then any road will take you there,” is apropos in project planning and implementation. You must identify the goals and outcomes that the new strategy or project should achieve. Without a clear picture of what you’re trying to attain, it can be difficult to establish a plan for getting there. Another often neglected aspect of goal setting is to account for variables that may hinder your project’s ability to reach those goals, and to lay out contingency plans. The better prepared you are, the more successful the implementation will likely be. 2. Determine Roles, Responsibilities, and Relationships. Once you’ve determined the goals you’re working toward and the variables that might get in your way, you should build a roadmap for achieving those goals, set expectations among your team, and clearly communicate your implementation plan, so all the key players know who the decision-makers are, who’s accountable for what, and what to do when an unforeseen issue arises. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 38 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 3. Define, and Assign Responsibilities for the Work. The project manager is key to successful strategy implementation from this point forward. Once you know what needs to be done to ensure success, determine who needs to do what and when. Using your project timeline and goal list, delegate tasks to the appropriate team members. 4. Execute the Plan, Monitor Progress and Performance, and Provide Continued Support. Now is the time to put the implementation into action. A central role of the project manager is to guide and support employees effectively in completing their assigned responsibilities. The project manager is also there to answer questions the team might have, or address challenges and roadblocks they may be experiencing. On-going monitoring and evaluation of the project progress using daily, weekly, and monthly status reports and check-ins to provide updates, re￾establish due dates and milestones, and ensure all elements of the project are aligned. 5. Take Corrective Action: Adjust or Revise, as Necessary. Implementation is an iterative process, so the work doesn’t stop as soon as you think you’ve reached your goal. Processes can change mid-course, and unforeseen issues or challenges can arise. Sometimes, your original goals will need to shift as the nature of the project itself changes. It’s more important to be attentive, flexible, and willing to change or readjust plans as you oversee implementation than it is to blindly adhere to your original goals. 6. Get Closure on the Project, and Agreement on the Desired Outcomes. Everyone on the team (including internal and external stakeholders) should agree on what project “success” should look like based on the goals set at the beginning. 7. Conduct On-going Assessment and a Retrospective Review of How the Project Went. On￾going assessment using a continuous improvement project management philosophy is essential to it’s success (see Step 5 above). Additionally, once a project has been fully implemented, as well as on a regular basis after that, look back on the project and evaluate how things went. Ask yourself questions like:  Did we achieve our goals?  If not, why? What steps are required to get us to those goals?  What roadblocks or challenges emerged over the course of the project that could have been anticipated? How can we avoid these challenges in the future?  In general, what lessons can we learn from the process? While failure is never the goal or desired outcome of any project, an unsuccessful or flawed strategy implementation can prove a valuable learning experience for an organization, so long as time is taken to understand what went wrong and why. So, take the lessons learned from the Tyler-Modria ODR pilot project, and use them to develop new and successful ODR programs to improve access to justice for all New Mexicans. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 39 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDICES APPENDIX 1: Online Survey Responses APPENDIX 2: AOC ODR Reports December 14, 2020 – June 30, 2021 APPENDIX 3: AOC-ODR Program Evaluation Survey Questionnaire APPENDIX 4: AOC-ODR Letter from Judge Levy and Josh Pando APPENDIX 5: The PRAXIS Project Consulting Team APPENDIX 6: Modria-New Mexico Courts ODR Poster APPENDIX 7: References APPENDIX 8: Research Articles and Reports for Reference PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 40 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 1: ONLINE SURVEY RESPONSES Presented Verbatim as Written by Survey Respondents – Annotated by Researchers in red italics. The researchers conducted content analysis on the responses to the survey, and organized the comments into the following distinct thematic categories: ● THEME 1: ODR Has Tremendous Potential to Improve Access to Justice in NM in a Cost-Effective Manner ● THEME 2: The Modria ODR Program has Been Ineffective and Inefficient ● THEME 3: Barriers to Effective Utilization of Modria: Ease of Use,Language, and Other Accessibility Features ● THEME 4: ODR Should Have a Video Component/Live Mediator Option ● THEME 5: Technical Issues with Modria Interface/Processes ● THEME 6: Difficulty Accessing Modria/Participation ● THEME 7: Debt and Money Due is an Inappropriate Case Type for Modria ● THEME 8: Inadequate Training/Communication – Confusion About the Program ● THEME 9: Issues and Concerns Regarding Legal Processes ● THEME 10: Time Constraints and Deadlines Impede the Settlement Process ● THEME 11: Need for Improved Planning and Implementation Strategy ● THEME 12: Some Positive Things! ● THEME 13: Miscellaneous Comments THEME 1: ODR Has Tremendous Potential to Improve Access to Justice in New Mexico in a Cost-Effective Manner PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 41 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● ODR offers the Judiciary an opportunity to further its goal to increase access to justice throughout New Mexico across economic, social, and geographic lines. This program didn’t work, but learn from it and grow ODR in things like traffic cases and compliance cases. ● Modria is a bad fit for the project. The platform was misrepresented as to purpose and capability, has serious design flaws, and works poorly. However, learn from this failure and develop new ODR programs and services that can increase access to justice for the people of New Mexico. ● If used for the right kinds of cases, and by using other communication platforms (e.g. Zoom), ODR can provide a cost-effective method of dispute resolution. ● Great idea and important to use technology to create a more efficient and faster way to resolve disputes – debt and money due cases and this platform were not the place to start. ● I think it is a good idea and is somewhat effective – needs a great deal of re-planning for improvement, different case applications ● ODR is a worthy attempt to help parties try to settle these types of cases more quickly and efficiently and even remotely, especially in a rural state like New Mexico; however, it's made more difficult without reliable technological access and a large enough caseload to support it. THEME 2: The Modria ODR Program has Been Ineffective and Inefficient ● There was a very, very low number of cases that settled in ODR, compared to the numbers that were referred. I think that if people selected the mediation option it would increase the number of cases that settle in ODR. ● It was not very successful in the 2-year pilot. ● I was excited to see this program start but it was a disaster. Please don't bring it back. ● I think this is a creative idea, but it needs some work. Our cases that were going to ODR normally go to our Arbitration program, which has a successful participation rate. Most of the cases that went to ODR did not settle/were not resolved, so it just took longer for them to get to our arbitration program, where the case was then taken care of. ● ODR has not proven to be cost effective. It is not making a significant difference in the debt/money due caseloads and has not proven to have value for either New Mexican consumer or AOC. During the pilot in the summer/fall of 2019, less than a dozen cases reached a negotiated agreement through the ODR platform ● We represented a creditor (public school district) owed money by a former employee. An attorney had entered an appearance but failed to answer. We filed for default, and that was when we received an invitation to the ODR program. At that point, we saw no benefit to ODR, as we were almost certain to be granted the default. We might have participated if we had been shown how the program could have been of benefit. ● It seemed like for the few cases where mediation was chosen, those had more settlements than cases where there was no mediation. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 42 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● I don't think the program is needed. Our office is repeatedly contacting the Defendants to try to negotiate a settlement and those attempts are well documented. Most Defendants have a DMC and if they are not part of the communication online a settlement is not really possible. ● ODR is not worth the money, time or trouble it has taken. ● Based on the data from AOC, very few cases have actually been "settled" while the cost has been significant. The program itself has not proven to be effective. ● Not cost effective ● The whole program is not efficient at all. it does not solve any cases at all. the majority of cases are either dismissed for lack of prosecution or go into default. THEME 3: Barriers to Effective Utilization of Modria: Ease of Use,Language, and Other Accessibility Features ● The Modria system is not user-friendly, and is not intuitive to use. ● Very confusing to use – reliance on email as a part of the process was not clearly explained, and caused unneeded delays in the process. ● Very frustrating to use. ● Language Accessibility ● Allow access to Spanish speakers ● ADA compliance not there ● Add multiple languages especially Spanish and Navajo ● Multiple Language availability ● Make it available for all languages. ● Did not provide ADA accessibility, in spite of claims by Modria ● Multiple language access ● Would be helpful if somehow Spanish speakers could be advised about using the system THEME 4: ODR Should Have a Video Component/Live Mediator ● Video vs text, live human interaction not just text ● Putting nothing into the people process (meaning mediators and participants) has FAILED. ● Transition to other technologies such as zoom/google meet, or telephone. ● Add a portal in the platform to allow a meeting in a video conference to discuss matter. ● effective communication - a phone call / video conferencing ● Video Mediation Module imbedded within Modria PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 43 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Needs a mediator ● Have a mandatory meeting via teleconference or phone to discuss matter as part of process. ● integrate video interpretation ● have mediator available to help ● integrate zoom mediation with live mediators in lieu of asynchronous chat ● there needs to be a real person facilitating these cases ● Have a hybrid model that includes face to face interaction with PAID mediators. ● Give the portal the ability to schedule a videoconference\ ● Use video conferencing ● Add an account manager or mediator to manage the process and contact parties. ● need to actually have contact with the other side ● Consistency of use of mediators. Inclusion. ● Highlight mediation as an option (cases that used mediators seemed to settle more often than those that did not). Make sure mediators have the ability to work within the program to submit documents, etc. ● Allow for telephonic participation in the program, for those parties without email or internet access. THEME 5: Technical Issues with Modria Interface/Processes ● Add the functionality for multiple defendants ● Streamline the registration process ● If there is more than 1 defendant, allow it for all defendants. ● The program did not always allow us to have private communications with the mediator. That is a bug that needs to be worked out. ● Allowing more flexibility in payments. Not requiring all payments to be equal. ● If a deal is reached the parties should be able to draft their own settlement documents as the ones used by ODR do not comply with all federal collection laws and or IRS notice standards ● Multiple defendants ● Allow for a monthly payment plan that extends over a certain number of months which specifies the court will keep the case on an inactive calendar (but not subject to dismissal for LOP) until the time for repayment ends. ● Make it easier to upload documents so in cases where there is a dispute everyone can look at the document just like they were doing an in-person mediation. This took a lot of time to resolve with tech support. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 44 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Ease of data loading with names, case numbers, etc. ● Post the answers to the complaints in ODR so the mediator can see all the pleadings to save time and frustration for the litigants. ● Received information that the attorneys were not able to add their paralegals to the system, and that firms were unable to add the email of the attorney working on that case. ● Problems with the ODR platform. We received many calls saying Tyler was not helping them and asking what they should do. If there is a way to have a Tyler team assigned for ODR platform questions, that would be helpful. I think having one person to troubleshoot ODR is very difficult. ● The process was long and added additional legal fees for Plaintiffs with little to no resolution. Make it easier for the Defendants to participate and more resolution will occur in my opinion. ● SMS notifications (when offers are made) ● We had no success with ODR. Not a single one of our files was successful with the ODR program. You need to be able to have more than 1 email associated with a file. ● Allow a repayment plan to be included - monthly payments instead of lump sum (If this is accurate, it is a deficiency in the program design – but if it is accurate, it demonstrates again that users need more training in the use of the system.) ● On the written notices provided, it was not clear who the parties were requiring us to look up the case numbers, etc. Those should include the caption. ● Each attorney receives 100's of emails per day and to find one from this ODR program to respond to is unrealistic. The program needs to be ran with a real person facilitating the agreements. ● We should be able to clearly state a demand for the court costs and attorney fees ● Clarify different channels of communications. Disputants don't seem to grasp the private/public channel options ● Many callers would call for assistance in creating a profile . Maybe access in adding/removing parties ● If dismissal is filed then Plaintiff must end ODR, court can email Margarita Terrell or Josh Pando to close/end ODR. Why can't the program be ended upon the Dismissal filed and docketed on the case. ● The name of the case, including the Plaintiff and Defendant, not just the case number, should be visible with each message and login. ● We should be able to to offer a discounted interest rate if the defendant is willing to make payments ● Clarify the options for what kind of agreements are acceptable such as stipulated agreement, etc. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 45 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Uploading documents was not always (usually) successful. ● multiple defendants should be allowed ● Allow the mediator to end ODR if the parties settle out of court. Resolving that took a huge amount of time. ● Add the ability to move back to previous screens in creating the agreement. ● Minimize the steps to make an initial offer. ● Add an option to edit the agreement prior to submitting it. ● Minimize the steps to make a counteroffer. ● Minimize steps to enter into a Settlement Agreement. ● Improved user interface ● Have attorneys enter the case into the ODR program not the clerks ● Ease of data loading ● I am a millennial and my overall experience with the platform is it was difficult to use, there were so many steps to log in to a single matter. There were tedious steps to enter an initial offer. The biggest downfall was getting the other side to participate, either because they were unaware of the program, or they did not understand how the platform works. ● include the full case name when you access the site ● Ease of resolution writing. More freedom to personalize outcome. Not all outcomes are “fill in the blank” ● Give the portal the ability to accept payment THEME 6: Difficulty Accessing Modria/Participation ● allow pleadings to be accepted even though ODR has not been completed. We are getting more and more stating they do not have internet access. ● ODR should not be referred until the defendant has responded. I referred many many cases after service had been effected, and ODR ended unsuccessfully due to a lack of participation. Most of those cases ended in default judgments. ● defendant / plaintiff participation ● Make more user friendly for use on cell phones ● DEFINE "participation" ● Making the program mobile friendly (like an app) would be helpful. We have lots of pro se customers who don't have computers but have smart phones. Also having kiosks in the computer that allow you to access your ODR portal (for those who don't have computers or cell phones) PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 46 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● provide confidential computer terminal access at a location in rural areas ● Make it more user friendly (accessible on phone, available in Spanish, easier platform, kiosks for the public) ● User friendly help kiosk center at each court ● ODR requires too much of Volunteer mediators and ill-informed participants. ● There were no formal means to re-refer a case to ODR. There were times when we had to reach out to Tyler directly to re-refer, because Plaintiffs were opting out of using the program. There was no tracking of these cases. ● In providing the pro se defendant's phone number or other contact information, we should be able to indicate that this is our last known information ● Kiosks at the courts ● Remove the Plaintiff's ability to end ODR, since the Supreme Court order indicates plaintiff participation is mandatory ● You should clearly state whether the plaintiff's voluntary election not to participate is without some form of penalty that might be imposed by the Court. ● Multiparty to be able to join the ODR program ● The program should not expect that the plaintiff will participate, if through counsel, without a fee. ● Call parties that have not replied to the initial offer ● Not sure - we didn't have many ODR deferrals in our court. However, many do not have computers or internet or just not enough knowledge to feel comfortable doing this through ODR ● Allow if no response from one party it get taken to court immediately ● The other side never responded when we used it, so make sure they actually respond ● I think you are on the right track. However, I live in the southeast part of the state and there are not many resources available for people dealing with the courts. When programs do become available, they often don't know about them. Many of the customers tend to be leery of what is being offered because they are nervous when dealing with the courts. Thank you. ● Not sure - we didn't have many ODR deferrals in our court. However, many do not have computers or internet or just not enough knowledge to feel comfortable doing this through ODR ● allow for more free form (not the same amount each time) in the online mediation agreement in the amount being paid page (starting at one amount and then increasing/decreasing during payment period) ● Promote funding access to public computer use in remote areas PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 47 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● require defendants to provide their current address, phone number and email ● we usually do not have e-mail addresses of parties ● There needs to be an incentive for plaintiffs to participate. Many plaintiffs just opted out or waited until the 30 days of ODR were over, and submitted their motions. THEME 7: Debt and Money Due is an Inappropriate Case Type for Modria ● Change the case type (traffic, warrants) ● ODR could be considered for traffic ticket cases. ● ODR has an adverse impact on the due process rights of defendants in debt and money due cases. There is an imbalance of power in these cases and using ODR exacerbates this imbalance. ● There is well documented abuse in the debt collection industry - see FTC reports and CFPB reports. ODR does not provide judicial review of these cases to guard against these problems. ● not the correct format for debt & money due cases ● Use it for a different type of cases ● Change the case type ● ODR in debt/money due cases has a lot of potential for abuse, and its use to date has not proven to be cost effective. Recommend the AOC discontinue the use of ODR in debt/money due cases ● Great idea and commendable efforts to try to make this program a success. Unfortunately, debt/money type cases are not optimal for online dispute resolution and no matter what changes are made, the cost will continue to outweigh its effectiveness. ● ODR is not appropriate for debt and money due cases and the program should NOT be continued as currently operated. ● This platform is not a good fit for debt/money due cases. It might be a better fit for traffic cases. ● AOC could improve the program by listening to consumer advocates who have raised concerns about the use of ODR in NM and specifically in debt and money due cases. ● Not for pro-se cases ● National research on ODR and experiences in other states raises similar concerns from consumer attorneys about problems with ODR in debt cases ● Use in traffic cases vs Civil being that some courts have in house mediation. ● not for pro se cases ● It was fruitless in our cases. Incurred additional attorney fees against pro se parties and did not resolve the matter. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 48 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● ODR Program creates additional work, on which, the vast majority of pro se defendants do not respond. Suggest limiting ODR program to only cases with answers filed. ● AOC could improve the program by stopping ODR in debt and money due cases. ● AOC could improve the program by doing a pilot in traffic ticket cases only. THEME 8: Inadequate Training/Communication – Confusion About Program ● The program was not effective at all. Once the case was referred by the courts and the parties were notified the case had been referred, the defendants would have problems logging in or finding their case. When I would tell them to contact Josh Pando, they would never be able to get through to him and then call the court back frustrated. Eventually the case would be removed due to lack of participation but only because the parties were not able to get any help. I referred the same case 3 times and the case never got resolved and the party never talked to Josh once. I even tried calling and emailing Josh several times to help the party and nothing ever happened. I even attended meeting to let Josh, Mateo, and Margarita know what was happening, but the meetings kept getting cancelled. ● Our clients have not been interested in use of the Online Dispute Resolution Program to date. (This suggests ineffective marketing and communication about the benefits of the program) ● I believe having training in general civil will give us more knowledge on answering calls. Most callers have other questions in regards to their case and we don't know how to answer/assist them. ● Call the opposing party after one of the first party has created the initial offer. ● More explanation to case participants. They don’t understand the process. ● Training on how and which cases may be re-referred to mediation. ● Better communication with the public when they are referred to ODR. ● There was confusion about what to do once the case received a non-participation. Should the case be re-referred to ODR? Should the case go onto the judges docket? Should the case go to our arbitration program? We had many plaintiffs who were just opting out of the system immediately. It didn't seem like there was any penalty for them, and therefore there was no incentive to actually try to use the ODR program. It also confused Defendants who wanted to use ODR when the plaintiffs just opted out immediately. ● The distribution of information about the program. A flyer or information sheet should be included with the Complaint and/or summons to be served on the defendant, and included with the form for the Answer to the Complaint. ● Provide a more detailed description of the program before parties are invited to participate. ● The defendants don't really understand the process. ● More training w/ Modria's portal. Maybe having more access to update information and etc. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 49 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Provide examples of what the program looks like when parties participate (e.g. screenshots, videos, etc.). ● Have a good contact number or information for the court. ● Many defendants thought that they were "answering" by participating in ODR, they did not understand that they actually had to submit an answer with the Court. I think the letter that goes out to the defendant should have an answer form attached and all defendants should be encouraged to answer, whether they are in ODR or not. Probably needs to be some clarification that participation in ODR is not an "answer". ● knowledgeable help desk( provide them with more information) to help end users ● Include information about the next steps in the court process (i.e., Pre-Trial Hearing, Mediation, and/or Trial) ● ODR court personnel do not share the knowledge to other court staff regarding ODR program (lack of communication) ● Provide information as to how the program can help all parties. ● Better communication with the attorneys on what ODR is. ● A lot of the people have emails addresses and ok with the facilitations happening telephonically, but they need a real person to talk to when they have issues or questions. The State needs to show good customer service if they are going to require these cases be referred to this program. The attorneys are not very responsive to the email system. ● thorough explanation of the process ● Have a good contact phone number for when parties have questions. ● More resources available for the public to understand so we won't have to spend so much time trying to explain it. ● Take steps to make defendants more aware of the program and better assist them through the process. ● If the case is assigned to another mediator after the first appointment was made, tell the first person so mediator number one does not waste his time trying to do the work. ● Training on understanding mediation process ● increased collaboration between court staff and ODR team ● Information about ODR included on court websites. It should be more user friendly, and easier to find, and provide step by step instructions for the entire process. ● There is inadequate information about whether defendants understand their rights through the ODR process. ● Include contact information for parties to contact each other and the Court's Mediation Department. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 50 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Advertise the availability of the program more effectively. ● The program directors need to be available to answer questions, answer their emails and responds to parties with questions ● Make citizens AWARE of the OLDR ● Advertise or inform the public that the program is available ● Educate the parties about the option when a complaint or answer is filed ● Provide a form of notice advising about the program to be served with the complaint ● Add a case manager/mediator to help the parties get in contact ● My practice is always, upon receiving a response to the complaint, to send a letter inviting the defendant to discuss some form of settlement/resolution without the matter proceeding much further. So, when I do this, and then this ODR program intervenes, it gets confusing. I prefer to send to my own letter because I can clearly indicate my goals as well as indicate to the defendant our willingness to discuss the Defendant's financial circumstances to allow me to fashion a remedy. The ODR program does not allow that flexibility. We are required to basically state a term of repayment and the amounts we want. ● Make sure people know about the program (when the case is referred, send ODR info sheet and answer information) ● better communication ● Input from the courts, to see the other side of the program. ● Explain the process to participants both in person and online. ● Make sure both parties are reached and notified of this process ● Ensure both parties participate and aware of the responsibilities ● Point out the possible advantages, and such notice, to reaching a quick informal resolution ● Better communication and information provided to judiciary employees, including clerks and judges and TCAAs. ● grow the AOC central communications personnel knowledge ● Make the portal more user friendly for those that struggle ● Provide more assistance to defendants ● Training material available to us the ones answering the calls the clerk telling people about the program. ● in-person training for individual courts ● Publicize the option with posters, fliers, and simple instructions on how to request the option ● Offer a CLE on the program so attorneys can become familiar with it before participating for the first time. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 51 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Ensure all parties understand the basis for the program ● Time to get familiarize with the concept. ● make it more known ● dedicated staff THEME 9: Issues and Concerns Regarding Legal Processes ● The mediator should be able to upload Judgments. My mediations have often been resolved via Judgment and the platform does not permit the mediator to upload it after the parties have reviewed and agreed. This makes no sense. ● Also, we received feedback from some attorneys that when they chose to use the mediation feature, they were not able to speak with the mediator one-on-one in the chat function or exchange documents with the mediator only. They said they were able to upload, but that the documents were there for the other party to see as well. ● We are not okay with agreements that are not signed off on by the judge. ODR does not allow that. ● Do not report the Plaintiff as failing to participate if exercise their option to opt out of settlement negotiations. Either don’t provide the option to withdraw or dont report as failure to participate. ● We had a plaintiff's attorney contact us twice, because a non-participation was generated, but he stated he participated in ODR mediation. ● Cases should only require ODR if answer has been filed. ● We should be able to seek the contract rate of interest ● If the defendant participates in ODR but does not reach an agreement, consider the Defendant as having filed an Answer. ● Allow for a Stipulated Judgment option, monthly payments but also attorney fees and court costs. Explain the stipulated judgment will be filed with the court and end the lawsuit provided payments are made. ● As an attorney for the creditor, we have very strict settlement guidelines, and we use those figures when putting in what would be possible. Having a Defendant propose a different amount is rarely approved without a documented severe and permanent hardship, and the back and forth is futile. ● easier enforcement when defaults happen ● Being able to email the Defendant directly to discuss settlement options and not having to use the Modria system at all would be preferred. A person from the Court's mediation program PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 52 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 could be copied on the email. Our collections team will have been calling the Defendants for months before the lawsuit is filed advising of settlement parameters so using Modria after the case is filed is redundant and time consuming for Plaintiff's counsel. ● Determine if the case is ripe for mediation. In my only interaction with the program the defendant had not answered, and a default was in process but there was no method to stop the mediation process. ● ODR should provide a consent judgment resolution option. ● Lawyers understanding they can’t “pressure” other parties. Responses delayed due to messaging ● A plaintiff should not be required to participate until the defendant has at least filed an answer or other response ● The settlement options available in the program. In order to appeal to plaintiffs, there should be an option to submit a stipulated judgment, albeit on an interim basis, pending payment, in case a defendant fails to comply with the terms of payment. ● Most Defendants have a Debt Management Company they are making payments to and that they have given a Power of Attorney to. The DMC rarely hires counsel and is not part of the Modria discussion so the case really can't be settled as they are holding the settlement funds. ● There is no monitoring to see if debt collectors are following Attorney General regulations and District Court rules, as well as the new federal regulations around the Fair Debt Collection Practices Act. ● attorney fees should be added ● It is not clear whether providing the defendant's phone number or other contact information (particularly electronic contact information such as email) can be done without liability such as under the TCPA. ● I do not understand why you would elicit what our lowest settlement figure would be. Often times that really depends on the course of the discussion and the defendant's payment ability/financial circumstances. It is not derived in a vacuum. ● No monitoring of time-barred debt, or other potentially false representations about a debt collector's ability to collect on debt. ● No monitoring of communications, including required disclosures under the federal Fair Debt Collection Practices Act. ● Making sure all judges know that once the case goes into ODR they should not rule on any motions. We saw plaintiffs filing for default judgments during the time the case was in ODR. ● Many times, the notice is sent before Defendant has even filed a responsive pleading. The timing of the notice is one possible area of improvement. ● Have the judges refer them? I don't know how that works right now. ● Front and back-end check for violations of federal and state debt collection laws PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 53 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● Front end check to ensure that plaintiffs are following court rules ● require defendants to disclose whether they have a debt management company and identify it if they do THEME 10: Time Constraints and Deadlines Impede the Settlement Process ● On occasion, I have needed more time to resolve a case. I am unable to ask for more time directly through the platform and must go through another person. The direct route makes more sense. For example, it took days for my administrator to get to my request for more time. In the meantime, the case expired and dropped out of my dashboard. I was very close to resolution, and it disappeared. ● In ODR, there was a 15-day deadline for the Defendant to file an answer once the case finished in ODR. This ended up being more strict/more difficult for Defendants than our normal arbitration process because with arbitration, as long as the Defendant filed an answer before the plaintiff filed for default, the case would go to arbitration. Sometimes the plaintiffs don't file for default for months, so this gives the Defendant a longer period of time to file an answer. ● amount of time. ● Give the mediators more time after sending the email they have been appointed. For those of us who still work, I do not check my work email in evenings or on weekends and I miss a deadline to post a welcome message. Thus, being appointed on a Friday with a deadline of Monday or Tuesday is not practical. ● if e-mail address are not available, case should be exempt ● shorter deadlines ● Mediator should have the ability to extend the time limit of the mediation if needed ● time effective for everyone involved THEME 11: Need for Improved Planning and Implementation Strategy ● The project appears to have been created to install Modria, rather than by creating a needs analysis, a vision, a mission statement, a strategic plan, a request for proposals, not starting on a small, manageable scale with sufficient time for development through experience, and with a rush to deploy – it was “look after you jump.” ● Listen to the input of consumer advocates and experts on ADR-ODR to improve ODR. ● This program was originally initiated by the Court and AOC without any input from creditor or consumer attorneys. There were significant problems in the original concept. Some changes were made in response to efforts by consumer attorneys to improve the proposed project. However, the ODR program has received significant negative comments and recommendations to cease use of ODR in debt cases. A meeting was held with AOC representatives and a member of the State Senate at which AOC made promises about providing data and reviewing the use of PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 54 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 the program. It has been difficult to get this information from AOC. The underlying problem - that is, using ODR for these types of cases - has not been addressed. ● The project needs to spur an initiative to thoughtfully consideration of how technology may be utilized to provide meaningful access to justice and judicial economy in a state with substantial poverty, limited resources, and far- flung geography. ● If the court wants to focus on the problem of defaults in these cases, ODR does not provide a solution. A pilot project using other strategies to encourage defendants to appear in court for a hearing on the merits should be explored. ● The project must be leveraged, not as a failure but, as a successful first foray into innovation. There are lessons to be learned. There is the discovery of the potential for how online services will enhance the administration of justice. ● Build from what has been learned – do not throw the baby out with the bath water. One must take wobbly first steps and fall a lot to learn to walk. There are no shortcuts. ● Leadership should utilize strategic and operational planning to improve effectiveness of ODR for various types of cases, and to improve access to justice in NM. Work should include a needs analysis, a vision, a mission statement, a strategic plan, a request for proposals, starting on a small, manageable scale with sufficient time for development through experience, and with adequate time to do it right. See https://www.aboutrsi.org/dme/dme-overview . ● The next step should not be how to fix Modria but, to take the time to take the development and design steps outlined here. Modria has had ample time to prove its worth. Time to move on. The flaws of Modria are too numerous to comprehensively list here. If Modria is to remain with the project, then a useability task force should be created to address the issues with the platform. ● The New Mexico leadership working on this project needs to be encouraged. New ground has been broken. Vast new potentials have been opened up. Give the good people working on this project the time, resources, and guidance to go forward. There is a rich opportunity to be harvested. THEME 12: Some Positive Things! ● First, what was not a problem: Good New Mexicans worked hard and long to make this project work. They need to be acknowledged and respected for their efforts. ● Open it to more cases than just debt (such as mediation between neighbors, etc.). ● I think it is a good idea and is somewhat effective – needs a great deal of re-planning for improvement, different case applications ● ODR is a worthy attempt to help parties try to settle these types of cases more quickly and efficiently and even remotely, especially in a rural state like New Mexico; however, it's made more difficult without reliable technological access and a large enough caseload to support it. ● The way the payments are calculated by the system is great. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 55 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● I do not normally work in the Civil Department. I have handled some instances where I have filed Prose responses and made sure the defendant's email address was in Odyssey. I used the proper event codes to initiate the ODR tab. I do know that when this program 1st began there were issues as to people not receiving their initial emails but were told to check their spam. ● Overall, I do think that it is an effective program. I have seen where the parties do not reach a settlement then the case will have to come before the court. However, I do believe this is a good program to have. THEME 13: Miscellaneous Comments ● Let us know what is going on ● Attorneys don't seem to take this program seriously in order to make the program work. ● Not enough personnel to fully operate the program to run effectively ● Multiple parties can be referred to ODR? ● ODR to date has failed. ● We were involved in every one of our files that went to ODR. Many times SOPA had us as non￾participants. ● The email address used for the program itself. The one used was much too long and complicated, and likely ignored by parties. ● needs everything explained ● if using an online platform for mediation the same type of platform should be used to submit documentation and to file ● Dedicated FTE to ODR (ADR staff & JID staff) ● This court has not had any cases that participated in the program. ● More flexibility on writing outcomes/results ● The process by which cases are referred to the program. It should be automatic in some way, and not rely on a clerk to remember to refer the case. ● Defendant/Debtor should be the one to make the first offer. ● I haven't actually participated in this program. My cases in it have either been exempted or otherwise resolved. ● Allow payments to be made on the platform. ● needs to move quickly ● the entire program was confusing from beginning to end. I would not like to see it revamped. ● needs to be reasonable PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 56 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 ● New Mexicans with exempt income or assets, or with a debt discharged in bankruptcy may be misled or intimidated into entering an improper settlement agreement ● interest rates should be at the rate set by contract ● both sides made to participate in good faith ● seemed to cause more issues than solutions ● Make ODR optional not mandatory ● needs to be fair ● needs to be what people can afford ● regular mediation resolved more cases in less time ● suggestions/opinions from the parties involved. ● PAY all mediators! ● Please don't bring it back. It didn't work. ● have someone monitor it ● This question presumes that it is appropriate to "grow" the program. I do not agree ● Offer an option through the clerk's office for ODR ● None at this time. It's a program that's still taking off. Most of our local collection clients just want to get a judgment and haven't seen tangible advantages of use of the system yet. It takes a few positive experiences with payment plans to get to that point. ● I think at this juncture, us mediators have tested and provided services for the program to demonstrate its effectiveness and that it is solving cases for the court. We deserve to be paid for our skills and services. ● Patience ● Work closely with the existing ADR/Mediation programs in the courts. ● Please pay attention to the comments provided. ● It was unfair on my part they dropped it after paying 77 to register case, money is scarce. The check was from the courts that the other party took out of someone else’s mailbox, cashed bond money and spent what wasn’t hers. ● Consider making it mandatory, similar to 2nd District's ADR program. ● Pay mediators ● Identify the neutrals in advance of committing to participation. ● Have a backup PAID mediation program to service those who can’t or won’t mediate online. ● make sure other side participates in good faith PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 57 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 2: AOC ODR REPORTS DECEMBER 14, 2020 – JUNE 30, 2021 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 58 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 59 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 60 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 61 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 62 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 63 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 64 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 65 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 66 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 3: AOC-ODR Program Evaluation Survey Questionnaire SURVEY QUESTIONNAIRE Think about your involvement with and participation with the AOC Online Dispute Resolution Program (ODR). How effective do you think the program is today, and how do you think it can be improved? NOTE: There are four (4) parts to this survey. A. Thinking about the online program (Modria – Tyler Technologies) being used for Debt and Money Due cases today, based on your expert opinion, please list ten (10) things that you think should be changed in order to improve the effectiveness of the program to successfully resolve cases? Ten distinct text boxes for responses B. From your perspective, what four (4) things would you recommend that the AOC do to improve and possibly grow the overall use of Online Dispute Resolution in New Mexico to better provide access to justice for its citizens? Four distinct text boxes for responses C. What additional comments and suggestions do you have regarding the Online Dispute Resolution program, that you think are important for the AOC to hear? D. Please list the names and contact information (email or phone number) for up to four other people who you believe would be helpful as participants in this study. NAME RELATIONSHIP TO ODR CONTACT – Email or Phone # PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 67 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 4: AOC-ODR Letter from Judge Levy and Josh Pando PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 68 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 5: THE PRAXIS PROJECT CONSULTING TEAM The PRAXIS Philosophy drives everything we do in providing management consulting and training. We believe that a consultant should act as an educator, helping the client achieve the level of self￾administration they are comfortable with. Controlled growth ensures quality service and longevity in relationships with our clients. We accept new clients only when our talents fit well with their needs and our staffing level is appropriate. SUMMARY OF PROFESSIONAL EXPERIENCE The project consultant team is Steven G. Meilleur, Ph.D. and Peter C. Meilleur, SHRM-SCP, assuming responsibility for performing each phase of the project. Together, both consultants have experience working with public, private for-profit, and private non-profit organizations and on multiple human resource consulting projects, including program and IT evaluation. Steven G. Meilleur, Ph.D., SPHR is the President/CEO of PRAXIS Management Solutions, LLC, and he has more than 40 years of professional experience in the public, private non-profit and private for-profit sectors. He specializes in all areas of human resource management, classification and compensation, strategic and operational planning, program research and evaluation, organizational research & development, training & development, non-profit organization management & board development, and management systems design. Dr. Meilleur has extensive experience working with public, nonprofit, and private for-profit organizations on multiple classification & compensation, organization development, general management, governance, and other human resource consulting projects. Both as an HR Executive and as a consultant, Steve has designed and helped to implement numerous classification and compensation programs in the public sector, the private for-profit sector, and the private non-profit sector. Dr. Meilleur is a Lecturer at the UNM School of Public Administration, teaching in the areas of human resource management, and organizational development. He has also been on the faculty at Webster University in the graduate studies programs since 1994. He received a BA in English Literature and Education from Bucknell University, an Executive MBA from the University of New Mexico, and his Ph.D. in Leadership and Organizational Learning from UNM. He received his certification as a Senior Professional in Human Resources (SPHR) by the Society for Human Resource Management in 1995. Peter C. Meilleur, SHRM-SCP is an Associate and research analyst with PRAXIS Management Solutions, LLC, and has 9 years of experience in research and analysis, HR consulting, risk management, training and development, and technology management. He is currently the Strategic Support Manager for the UNM College of Nursing. Mr. Meilleur graduated Summa Cum Laude with Highest Honors from California Lutheran University with a BA in Communications. He was certified as a Senior Certified Professional from the Society for HR Management in 2021 (SHRM-SCP). PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 69 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 6: MODRIA - NEW MEXICO COURTS ODR POSTER PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 70 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 7: REFERENCES Adkins, A. (2016). Employee engagement in U.S. stagnant in 2015. Gallup Inc. Retrieved from http://www.gallup.com/poll/188144/employee-engagement-stagnant-2015.aspx American Bar Association Commission on the Future of Legal Services. (2016). Report on the Future of Legal Services in the United States, Recommendation 5.4. Retrieved from: https://www.americanbar.org/content/dam/aba/images/abanews/2016FLSReport_FNL_WEB.pdf. Batten, B. (2020). Online dispute resolution: The future is now. Canadian Arbitration and Mediation Journal, Vol. 29, No. 1 Spring 2020. Council of Europe. (2021). Online dispute resolution mechanisms in civil and administrative court proceedings. Strasbourg, France: Council of Europe. Gardner, H. (1985). The mind’s new science: A history of the cognitive revolution. New York, New York: Basic Books. Grønkjær, M., Curtis, T., de Crespigny, C., & Delmar, C. (2011). Analysing group interaction in focus group research: Impact on content and the role of the moderator. Qualitative Studies, 2(1), 16–30. Retrieved from https://doi.org/10.7146/qs.v2i1.4273 Heil, J. (1983). Perception and cognition. California: University of California Press. Kanter, R. and Spencer, L. (1989). Winning through participation: Meeting the challenge of corporate change with technology of participation. Dubuque, IA: Kendall/Hunt Publishing. Krenzke, T., Mohadjer, L., Li, J., Erciulescu, A., Fay, R., Ren, W., Van de Kerckhove, W., Li, L., & Rao, J. N. K. (2020, April). Program for the international assessment of adult competencies (PIAAC). PIAAC. Retrieved from https://nces.ed.gov/surveys/piaac/state-county-estimates.asp Krueger, R.A. (1989). Focus groups: A practical guide for applied research. Newbury Park, CA: Sage Publications. Kulp, H. and Schmitz, A. (2020). Real feedback from real people: Emphasizing user-centric designs for court ODR. University of Missouri School of Law: Dispute Resolution Magazine|Vol 26, No 2, pp. 6-12 Lacey, C. (1970). Hightown Grammar: The school as a social system. Manchester, UK: Manchester University Press Lewin, K. (1936). Principles of topological psychology. New York: McGraw-Hill. Mannes, M. (1990). The perceptions of human service workers in planning for the implementation of the family preservation services innovation in indian child welfare settings. Unpublished doctoral dissertation, Cornell University, Ithaca, New York. Marshall, C. & Rossman, G. (1989). Designing qualitative research. Newbury Park, CA: Sage Publications. Merriam, S. B. (1988). Case study research in education: A qualitative approach. San Francisco: Jossey￾Bass. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 71 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 Michigan Supreme Court, Office of Dispute Resolution. (2020). Using zoom to conduct online mediation: considerations and resources for community dispute resolution program centers. Retrieved from Community Dispute Resolution Program Miles, M. & Huberman, A.M. (1984). Qualitative data analysis: A sourcebook of new methods. Newbury Park, CA: Sage Publications. Miller, K. (2020). A manager’s guide to successful strategy implementation. Harvard Business School Online: Cambridge, MA. Retrieved from https://online.hbs.edu/blog/post/strategy-implementation￾for-managers Morgan, D.L. (1988). Focus groups as qualitative research. Newbury Park, California: Sage Publications. National Center for State Courts. (2015). The landscape of civil litigation in state courts. Retrieved from https://www.ncsc.org/~/media/Files/PDF/Research/CivilJusticeReport-2015.ashx. National Center for State Courts ODR Project (n.d.). Eight lessons to consider for ODR implementation. Retrieved from https://www.ncsc.org/odr/guidance-and-tools Parker, A., & Tritter, J. (2006). Focus group method and methodology: Current practice and recent debate. International Journal of Research & Method in Education, 29, 23–37. Rule, C. (2015). Technology and the future of dispute resolution. Dispute Resolution Magazine 21 (2015): 4, 6. Retrieved from http://law.scu.edu/wp-content/uploads/Rule-Technology-and-the-Future￾of-Dispute-Resolution-copy.pdf. Skeggs, B. (1997). Formations of class & gender: Becoming respectable. Thousand Oaks, CA: Sage Publications Inc. Soleymaniha, A. (2021). Transformational process design for dispute resolution. Canadian Arbitration and Mediation Journal, Vol. 30, No. 1 Spring 2021. Van Arsdale, S. (2016). User protections in online dispute resolutions, 21 Harvard Negotiation Law Review Vol. 107, Cambridge, MA. Wahab, M., Katsh, E., and Rainey, D. (eds.). (2021). Online dispute resolution: theory and practice. Second Edition. The Hague: Eleven International Publishing. Retrieved from https://www.worldcat.org/title/online-dispute-resolution-theory-and￾practice/oclc/1268320953?referer=di&ht=edition Weimer, D., and Jonas, P. (1997). Strategic planning: A participative model. Cardinal Stritch University, Milwaukee, WI. Witwer, A.R., Langton, L., Banks, D., Woods, D., Vermeer, M.J.D., Jackson, B.A. (2021). Online dispute resolution: perspectives to support successful implementation and outcomes in court proceedings. The Rand Corporation, Santa Monica, CA. PRAXIS Management Solutions, LLC 316 Encantado Ridge Court ● Rio Rancho, New Mexico 87124 [email protected] ● (505) 401-0942 FINAL REPORT: ODR Program Evaluation and Recommendations – NM Administrative Office of the Courts Page 72 of 72 PRAXIS Management Solutions, LLC Submitted 14 February 2022 APPENDIX 8: RESEARCH ARTICLES AND REPORTS FOR REFERENCE The following research articles and reports provide recommendations concerning the design of Online Dispute Resolution programs, and are attached for reference (listed in no particular order):  ODR for Courts (2017). Joint Technology Committee of the Conference of State Court Administrators (COSCA), the National Association for Court Management (NACM) and the National Center for State Courts (NCSC).  A Blueprint for Online Dispute Resolution System Design (2018). University of Missouri School of Law Scholarship Repository.  Recommended Best Practices for Online Dispute Resolution Service Providers. (n.d.). American Bar Association Task Force on Ecommerce and ADR.  Asynchronous ADR: Tipping In-Person Processes Off Their Pedestal. Bhalla, M. (2021). Canadian Arbitration and Mediation Journal, the ADR Institute of Canada.  Online Dispute Resolution: Designing New Legal Processes for Cyberspace. Katsh, E. (n.d.). National Center for Technology and Dispute Resolution.  User Protections in Online Dispute Resolution. Van Arsdale, S. (n.d.). The Harvard Negotiation Law Review  Virtual Interpersonal Community Mediation. Mawn, D.G. (2021). National Association for Community Mediation Think Tank Report.  Online Dispute Resolution Mechanisms in Civil and Administrative Court Proceedings. Council of Europe. (2021). Adopted by the Committee of Ministers of the Council of Europe.  Paths Forward for Online Mediation. Mosten, F. and Melamed, J. (2021). Final Report of the Mediate.com Online Mediation Training Task Force.  Strategic Planning: A Participative Model. Jonas, P. (1997). Cardinal Stritch University – ResearchGate.  Transformational Process Design for Dispute Resolution. Soleymaniha, A. (2021). Canadian Arbitration and Mediation Journal, the ADR Institute of Canada.  Using Zoom to Conduct Online Mediation: Considerations and Resources for Community Dispute Resolution Program Centers. (2020). Michigan Supreme Court Office of Dispute Resolution.