2 thoughts on “Hawaii’s HOPE Program Gets a Critical Review

  1. The conclusion you quote from the Lattimore et. al paper goes far beyond the data. Two commentaries published in the same issue of C&PP point out that:

    (1) Many other studies of SCF programs show good results: not only the original Hawaii program but also 24/7 Sobriety in South Dakota, Swift and Certain in Washington State (a glowing evaluation of which appears in the same issue as the Lattimore et al. paper), and Swift in Texas. Pilot programs using SCF principles to reform the prison-discipline systems in Washington State, Ohio, and Pennsylvania all seem to be working well, though formal evaluations have yet to be published.

    (2) The experimental set-up imposed a rigid formula modeled after the Hawaii program on four districts with vastly different cultures, institutions, and circumstances.

    (3) The model imposed by BJA used only jail as a sanction. Later implementations have had success in encouraging compliance with non-jail sanctions, and with rewards.

    (4) The sanctions used (averaging more than 10 days per violation) were far more severe than those used in other, successful implementations.

    (5) Some jurisdictions jailed people for not paying probation fees, which was never part of the Hawaii model and obviously violates the principle of fairness.

    (6) Imposing a rigid program design prevented the process of consultation with stakeholders which is crucial to the success of any program.

    (7) Rigid program design also meant that the programs under study couldn’t be modified in the face of operational problems, but had to be carried forward according to the experimental protocol.

    (8) Every SCF program takes time to build credibility with those being supervised, so best practice is to roll a program out and shake it down first, and start evaluation later. This evaluation started on Day 1. As expected, the first batch of entrants had much worse outcomes than subsequent batches; the bad result reported was the average.

    (9) One dramatic success of SCF programs is in reducing drug use, but the evaluation did not even measure drug use as an outcome, and therefore couldn’t consider the benefits of reduced drug use in its benefit-cost analysis.

    If the question addressed was “Is HOPE a magical program-with-a-manual that will succeed everywhere if mindlessly replicated?” Lattimore et al. show that the answer is “Of course not.” But that’s not a surprise to anyone who has been doing this work.

    The right conclusion to draw from all the available data is that systems of swift, certain, and fair incentives (rewards as well as sanctions) can and do succeed when implemented with consultation and in a form consistent with local conditions. One key is minimizing the use of jail as a sanction. Programs mismatched to local conditions, or which violate the principles of fairness and procedural justice (including demonstrated goodwill toward participants) perform less well.

    SCF is an operating concept for corrections, not a specific program. How to fit that concept to local conditions is always a complex problem. But unless and until someone produces a logical argument in support of “slow, random, and arbitrary,” there is no reason to ignore the growing evidence that swift, certain, and fair tends to lead to better outcomes.

  2. Response to the comments of Mark Kleiman in regard to “Outcome Findings from the HOPE Demonstration Field Experiment” by Lattimore, MacKenzie, Zajac, et al. Criminology and Public Policy, Vol. 15, 4, 1-39.

    Posted by Pamela K. Lattimore (lattimore@rti.org)
    (1) Many other studies of SCF programs show good results: not only the original Hawaii program but also 24/7 Sobriety in South Dakota, Swift and Certain in Washington State (a glowing evaluation of which appears in the same issue as the Lattimore et al. paper), and Swift in Texas. Pilot programs using SCF principles to reform the prison-discipline systems in Washington State, Ohio, and Pennsylvania all seem to be working well, though formal evaluations have yet to be published.
    Response: The purpose of the DFE was to provide a rigorous test of the Hawaii HOPE model, based on the principles of HOPE espoused by Alm, Kleiman, and others. The intent was to determine whether the program and the findings purported in Hawaii could be replicated in multiple sites in the continental United States. Hawaii HOPE was described as a program using SCF principles. The DFE findings are based on a ‘gold-standard (RCT)’ test of the HOPE model; these findings were submitted, reviewed, accepted, and published by a reputable scientific journal. The South Dakota study involved breathalyzer testing of DUI offenders—a completely different target population and approach than the HOPE model. We have no comments on unpublished findings and will only note that to our knowledge the original Hawaii study was never published in the peer-reviewed literature and that the Washington State study involved a pre-post design in which the “pre” comparison group was identified and followed prior to the legalization of marijuana and the “post” treatment group was identified (largely) and followed (completely) after the legalization of marijuana suggesting different legal landscapes for the two study groups.
    (2) The experimental set-up imposed a rigid formula modeled after the Hawaii program on four districts with vastly different cultures, institutions, and circumstances.
    Response: To establish a program as ‘evidence based’ requires that the structure of the program be known and replicable. The principles of HOPE are straightforward—to hold individuals accountable to their conditions of supervision (no drugs, comply with office visits, community service, etc.) and impose short jail stays (quickly) if individuals violate those conditions, provide the opportunity for treatment if individuals repeatedly fail their drug tests, and revoke only after repeated violations. The HOPE model has been promoted to multiple jurisdictions across the United States and elsewhere on the assumption that it would be appropriate regardless of culture, institutions, and circumstances. Technical assistance was provided to the sites to assure their compliance with the HOPE model.
    (3) The model imposed by BJA used only jail as a sanction. Later implementations have had success in encouraging compliance with non-jail sanctions, and with rewards.
    Response: Although confinement was the prescribed response to a violation (per the Hawaii model), jail was not the only response in the DFE sites. A total of 3,550 sanctions were imposed on the HOPE subjects. Of these about 84% included confinement. To our knowledge there have been no RCTs comparing incarcerative and non-incarcerative sanctions so there is no evidence the use of these sanctions have had “success in encouraging compliance” much less greater success than jail.
    (4) The sanctions used (averaging more than 10 days per violation) were far more severe than those used in other, successful implementations.
    Response: The 10 days per violation is incorrect. The median jail stay in response to a violation across all four sites was 4 days and the site-specific medians were 2, 3, 5, and, 7 days. Mean jail days were 4, 5, and 6 (2 sites) days. These figures are for more than 2900 jail stays and would include increasing stays for multiple violations (escalation). Also, note that the four sites varied in the mean and median lengths of jail time used to sanction noncompliance, providing a test of the impact of length of short duration jail stays. Since we found similar results for all four sites the results do not support Kleiman’s conclusions and we know of no other rigorous scientific test of the impact of differences in (short) jail-days.
    (5) Some jurisdictions jailed people for not paying probation fees, which was never part of the Hawaii model and obviously violates the principle of fairness.
    Response: The HOPE model was to assure compliance with conditions of supervision. Payment of fees and fines is often a condition of supervision and was in at least some of the DFE sites. Whether holding someone accountable to this condition is “fair” is a normative judgement. However, it is essentially a moot issue relative to the DFE evaluation—59 of 3,569 (1.65%) violations by HOPE program participants (54 participants) were for failure to pay fees/fines. 134 evaluation participants had a fee/fine violation (54 HOPE probationers and 80 PAU probationers; all but 16 of these were in 2 of the 4 sites).
    (6) Imposing a rigid program design prevented the process of consultation with stakeholders which is crucial to the success of any program.
    Response: Local systems planning was in fact a component of the HOPE implementation. One site had been operating a SWIFT program for more than a year and was thus onboard with the HOPE model. Two of the other sites had established statewide commissions that had been studying criminal justice and sentencing reform that were instrumental in applying for the HOPE implementation grant. Thus, HOPE was not “imposed” on these sites—they entered into the DFE freely and participation in it was pursuant to their own interagency criminal justice planning efforts. If we do not complete rigorous scientific studies how are we going to learn whether programs are successful in reaching the desired goals. Jurisdictions were provided with technical assistance throughout the program and stakeholders were involved in these meetings. Advocacy without rigorous scientific methods will keep us replicating programs that “feel good” to advocates without enabling us to be successful in obtaining the desired outcomes.
    (7) Rigid program design also meant that the programs under study couldn’t be modified in the face of operational problems, but had to be carried forward according to the experimental protocol.
    Response: There were modifications to a point. One site did not use dedicated HOPE probation officers (POs). In one site, the HOPE POs were responsible for all aspects of HOPE; in other sites, some tasks such as assessment, drug testing or warrant servicing were handled by specialized units. In one site, they chose not to use same-day arrest but gave the probationer a chance to turn him/herself in. These variations adapted to local needs, were documented in the implementation study, and again allowed for some variation in the program implementation as we assessed within site findings—which were similar.
    (8) Every SCF program takes time to build credibility with those being supervised, so best practice is to roll a program out and shake it down first, and start evaluation later. This evaluation started on Day 1. As expected, the first batch of entrants had much worse outcomes than subsequent batches; the bad result reported was the average.
    Response: Enrollment in the DFE ranged from 17 to 26 months across the four sites. Thus, any portion of the study populations that could be ascribed to a “shake down period” would only be a small portion of the total enrollment. If we compare the HOPE and PAU cases who were enrolled in the first 3 months of the evaluation, 49% of the HOPE participants compared with 54% of the PAU probationers had a new charge (not significant, p = 0.40). The failure rates for those enrolled after the first 3 months, were lower—for both groups: 38% for HOPE and 42% for PAU (not significant, p = 0.16). The reasons for these findings are (at least) two-fold. First, at DFE startup, sites pulled from their ‘stock’ probation populations focusing on individuals with recent violations—these individuals were “riskier” than a new incoming medium/high risk population that provided more of the cases as enrollment continued. Second, the earlier enrollment cohort, by definition, would have had a longer time period over which to fail. So there is no reason to believe that the “early” start of the evaluation had an impact on the overall findings of the DFE.
    (9) One dramatic success of SCF programs is in reducing drug use, but the evaluation did not even measure drug use as an outcome, and therefore couldn’t consider the benefits of reduced drug use in its benefit-cost analysis.
    Response: The evaluation is looking at drug use and findings will be published in subsequent articles. Analysis of the results from the random drug testing of HOPE participants does show a decline in drug use during participation in HOPE (based on agency and fidelity records) and a reduction for HOPE compared with PAU (based on oral swab drug tests conducted in conjunction with follow-up interviews). It is not clear how to consider the “benefits of reduced drug use in its benefit-cost analysis” as we were not looking at individual health outcomes but systems saving associated with reduced criminality. More details will be forthcoming in a future article and in the HOPE DFE evaluation final report.
    (10) If the question addressed was “Is HOPE a magical program-with-a-manual that will succeed everywhere if mindlessly replicated?” Lattimore et al. show that the answer is “Of course not.” But that’s not a surprise to anyone who has been doing this work.
    The right conclusion to draw from all the available data is that systems of swift, certain, and fair incentives (rewards as well as sanctions) can and do succeed when implemented with consultation and in a form consistent with local conditions. One key is minimizing the use of jail as a sanction.
    Response: There have been six RCTs testing principles of SCF/HOPE in different jurisdictions and with somewhat different populations—five of those found no differences in recidivism outcomes between those in the HOPE/SCF programs and probation as usual. There is absolutely no evidence that we are aware of that supports the contention that “One key is minimizing the use of jail as a sanction.” As noted above, jail stays were “short” by any consideration.
    Further, there are numerous examples as HOPE was promoted of the role of short jail stays as integral to HOPE, including the following quotes from a 2013 PBS interview with Judge Alm (http://www.pbs.org/newshour/bb/law-july-dec13-hawaiihope_11-24/) in which he highlighted the role of jail as a sanction (along with principles of swift and certain):
    “Swift and certain is what’s gonna get people’s attention and help them tie together bad behavior with a consequence and learn from it.”
    “So I’m going to reflect that by only giving you a couple days in jail. And so, we’re gonna work with you on that.”
    Thus, it is hard to square the description of the role of short jail stays in the HOPE model as described by Judge Alm with Kleiman’s assertion of “minimizing the use of jail as a sanction.”
    (11) Programs mismatched to local conditions, or which violate the principles of fairness and procedural justice (including demonstrated goodwill toward participants) perform less well.
    Response: There is no evidence that the DFE programs were “mismatched to local conditions” or that they “violated the principles of fairness and procedural justice.” All four sites were committed to following the model and did so.
    (12) SCF is an operating concept for corrections, not a specific program.
    Response: SCF may be an appropriate “operating concept for corrections” but the current evidence base provides little support that it reduces recidivism.
    (13) How to fit that concept to local conditions is always a complex problem. But unless and until someone produces a logical argument in support of “slow, random, and arbitrary,” there is no reason to ignore the growing evidence that swift, certain, and fair tends to lead to better outcomes
    Response: There are multiple reasons to support “swift, certain, and fair” supervision principles over “slow, random, and arbitrary” but there is not “growing evidence that swift, certain, and fair tends to lead to better outcomes.”

Leave a Reply

Your email address will not be published. Required fields are marked *