Friday, June 27, 2014

Some Good News in Who Benefits from Family-Strengthening Programs


Whatever your political views, you likely share in concerns many hold over the difficulties facing socially and economically disadvantaged families in the U.S. But can the government do anything to directly help such families through family-strengthening efforts? Despite all you might have heard to date, there is some good news emerging from recent studies and my goal here is to describe that news.     

The U.S. Administration for Children and Families, specifically the Office of Family Assistance (OFA), is currently invested in three specific components of family strengthening, including efforts to a) improve the quality and stability of the relationships of couples with children, b) increase father involvement for those with fragile relationships with their children, and c) increase the quality of co-parenting between adults who have children in common but who are no longer in an ongoing relationship. My focus here is on research on the first of these types of efforts, as there has been considerable attention to the results of two major federal evaluations of programs aimed to help couples with low incomes and other disadvantages.

The large, multi-site studies were called Building Strong Families (BSF) and Supporting Healthy Marriage (SHM). I believe these studies received so much attention, in part, because they were connected with the somewhat controversial, government-funded efforts over the last twelve years to provide relationship education in support of the goals in the welfare reform law of 1996 to promote two-parent families. These two studies produced disappointing results, which have been lamented among those who support such efforts and trumpeted by those who are critical of such efforts. I think the trumpets have mostly carried the day.

But there is some good news for supporters of relationship education in recent findings, including within those two federal studies. Specifically, it is becoming clear that ethnic minority couples benefit at least as much or more than other couples from such programs. Some have suggested that this would not be the case because such programs were originally designed and tested with mostly middle-class, white couples.[i] Further, there is newly published evidence that, within the BSF study data set, the participants who were the most socially and economically disadvantaged benefitted the most in terms of impacts on relationship quality.

Before describing in more detail what I see as relatively encouraging, I will first describe a bit of background on these important federal studies and their findings to provide context for those who know little about them.

Building Strong Families (BSF) was a study of unmarried couples in the transition to parenthood, whereas Supporting Healthy Marriage (SHM) was a study of married couples. Both projects focused on couples at lower incomes, generally under 200 percent of the poverty line. In both cases, couples in multiple cities were randomly assigned to either receive or not receive a substantial program of relationship education and other couple support services, all designed to strengthen these families with regard to couples’ relationship quality, child outcomes (including father involvement), and stability. While there are some disagreements about the results, it is fair to say that the overall evidence suggested largely non-significant (BSF) or small (SHM) program impacts.

As a way to provide a bit more detail, allow me to give two brief summaries of the findings, as if from the perspectives of people who were either more or less encouraged by the results.  

             Less encouraged. The overall findings for BSF showed no evidence of positive impacts on couples’ relationships and father involvement. There were even some modest negative impacts in some sites. While there was a modest positive impact on child behavior, there were no other overall positive effects on child wellbeing. For SHM, while there was a range of statistically significant positive impacts, the impacts were modest. Further, these particular programs were expensive—much more so than most all historical efforts involving relationship education. In both cases, there was precious little evidence of positive impacts related to child outcomes. There were some very tiny positive impacts for children in the SHM study, but tiny is tiny.

             More encouraged. In the BSF study, one site (Oklahoma) did a particularly impressive job of getting couples in and through services—far outperforming other sites in this respect. The other sites did not, for the most part, get a lot of couples through much of the planned services. Across the whole study, only 55 percent of couples attended any of the relationship education services. Only Oklahoma demonstrated a range of significant and positive impacts on couple relationship outcomes at the 15-month assessment.[ii] While these impacts faded at the 36-month assessment, the children born to couples in the program group were 20% more likely than children born to couples in the control group to have lived continuously with both parents until that 3-year point—also, only in the Oklahoma site.[iii] In SHM, couples showed statistically significant gains at the 12-month assessment and these gains, while small, were largely maintained out to the 30-month assessment. In a field where most policy evaluations of social programs show no significant, lasting impacts, some see this as promising even as the need for improvement is obvious.[iv]

That’s the skinny version of what happened. There are detailed reports and endless commentary on the internet, if you want more information.[v] The dominant story across the media about these studies is that nothing worked. However, there was some good news in how SHM sites learned from experiences in the BSF study, and thus achieved far greater participation and follow-through among couples than BSF sites did. That is, by including strategies to reduce barriers to participation and reinforce attendance, SHM enabled more disadvantaged couples to attend a substantial amount of program services. That is encouraging.

Before moving on, I should mention that there is a whole world of research on relationship education that I am not attempting to cover here, with studies showing generally stronger, positive effects. Further, some experts in that field have been dismayed that so much attention has been focused on BSF and SHM in recent years. I will, however, retain a similarly narrow focus in order to cover findings related to social and economic disadvantage.

The Response of the Most Disadvantaged Couples

To my knowledge, with one exception, the only analyses done to date on the BSF data set have been conducted by the professional evaluation team hired by the Administration for Children and Families to conduct the study. The exception lies in analyses conducted by Paul Amato at Penn State. Amato has approved access to the BSF data set out to the 15-month assessment point, and he has just published a paper with important analyses from that data set.[vi]

Amato sought to assess whether couples at greater disadvantage received more, similar, or less benefits in BSF than other, less disadvantaged couples did. He analyzed outcomes related to both relationship stability (whether couples broke up) and relationship quality.  His method, which was different from any other analysis I have seen to date in this field, was to create a “disadvantage index” based on eleven factors in order to assess whether having a high or low score on this index affected how much couples benefited from the programs. I will quote from Amato’s paper regarding the list of factors going into this index (p. 347):

(a) the mother was less than 20 years old,
(b) the father was less than 20 years old,
(c) the mother did not have a high school degree,
(d) the father did not have a high school degree,
(e) the father was unemployed at baseline,
(f) the father earned less than $10,000 in the last year,
(g) the mother received public assistance in the last year (TANF, food stamps, Medicaid, SCHIP, SSI, SSDI, or WIC),
(h) the mother had one or more children from a previous relationship,
(i) the father had one or more children from a previous relationship,
(j) the mother or father reported no one to care for the baby in an emergency (excluding the partner),
(k) the mother or father reported no one to borrow money from in an emergency (excluding the partner). I [Amato] omitted mothers’ income from the index because the majority of mothers were not in the labor force.

While I need to skip over some technical detail here, I want to note that Amato approached the analyses in a particularly robust way. He tested his findings for what we call sensitivity to various specifications. Essentially, he tried the analyses with various sites left in or out and with various indicators of disadvantage in or out. The findings were robust across such tests.

Before testing for the program impacts, Amato found what one would expect: “The relationship risk variable revealed that higher disadvantage scores were associated with less support and affection, more destructive conflict, less constructive conflict, less trust in partner, more intimate violence, and lower overall relationship quality” (p. 350). Thus, the risk index captures risk as it was designed to do. The important question is, did their level of disadvantage matter for how much benefit couples received in the BSF intervention model? While disadvantage level did not matter for how the program affected couples’ stability (their odds of breaking up), when it came to relationship quality, those with more disadvantage received the most positive impact from the programs.

Quoting from Paul Amato’s paper: 

One of the major criticisms of BSF programs for unmarried couples (and federally funded marriage education programs for low-income couples) is that educational interventions are not effective for disadvantaged populations (Johnson, 2012; Karney & Bradbury, 2005). It is reasonable to imagine that poor couples are so overwhelmed by financial problems and everyday stress that they are unresponsive to relationship education programs and see them as largely irrelevant to their lives. If this were the case, then the most disadvantaged couples – those most at risk of relationship problems – would receive the least benefit from programs like BSF. This study, however, suggests the opposite: Contrary to the notion that disadvantaged couples do not benefit from relationship education, these couples may be the main beneficiaries of these services, provided that they are able to keep their unions intact. (p. 353)

Keep in mind, these results are for the 15 month follow-up. It is possible that if the same analyses are one day repeated for the 36 month follow-up, this same result would not be found. It is not unusual in this field to find impacts in an earlier period that fade by the time a later follow-up is conducted.

In contrast to these encouraging findings from Amato’s paper, results from a meta-analysis working its way toward publication suggest that the very poorest couples receive the least benefit from such programs. (I have the author’s permission to mention what I know about the analyses.)  I believe, however, that the type of analysis in this other study is far less sensitive to addressing the question Amato tested. Nevertheless, the findings from this other study align more closely with the arguments made by the researchers noted in Amato’s quote above, who have suggested that severe economic hardship may interfere with couples’ ability to benefit from such efforts. It is not hard to imagine that chaos and stress would interfere with learning new strategies in one’s relationship. On the other hand, when studies in this field do analyze whether impacts vary based on levels of prior risk, those at greater risk often get the most benefit. There is a lot of complexity here for researchers in the field to sort out.

Amato’s analyses are serious and thoughtful, and he obtained a potentially important finding that is not at all evident from the primary analyses conducted with the BSF data set. That takes nothing away from the main results in BSF (pooled across sites) that are legitimately disappointing for reasons about which serious people may not agree. But Amato’s analyses are encouraging, and perhaps even provocative, for suggesting that such services may actually provide the most benefit, on average, to couples with some of the greatest disadvantages in life. In fact, Amato goes so far as to imply that if the BSF study had recruited substantially more disadvantaged couples, the overall findings across the study would have been positive (p. 353).  

Amato’s findings are not unprecedented. They are the most sophisticated version of a type of finding that has been obtained before, wherein those who are more disadvantaged receive at least as much, and sometimes more, benefit from relationship education services than others.[vii] Amato notes that this is generally the case for various social programs (p. 353). What he found is also consistent with other studies focused on family strengthening that find positive impacts for programs given to highly disadvantaged couples and families. For example, Phil and Carolyn Cowan and their colleagues have demonstrated positive impacts from a program focused on father involvement in a study with low-income families, with a particularly large representation of Mexican American families (67 percent of participants). They found significant, positive impacts on couple relationship quality, father engagement, and children’s problem behaviors.[viii]

The Response of Ethnic Minority Couples

On to other encouraging news I want to share. In both the BSF and SHM studies, the evaluators were able to examine if the minority group with the largest representation got more or less impact than other couples. The largest minority group in BSF was African-American couples, and the largest minority group in SHM was Hispanic/Latino couples. (Because of the nature of the studies and the program sites, there was a relatively small percentage of Hispanic/Latino couples in BSF and a relatively small percentage of African-American couples in SHM; hence, the analyses for differential impact focused on the larger groups within each study.)

For the earlier assessment points in both BSF and SHM (15 and 12 months, respectively), there was evidence that the minority couples in the intervention groups received more benefit than other couples. That is, in BSF, African-American couples benefitted more than other couples.[ix]  In SHM, Hispanic couples benefitted more than other couples.[x]  I do not wish to exaggerate these findings in any way, but the pattern was found in both studies. However, the pattern did not hold up at the longer-term assessments in either study (36 months for BSF and 30 months for SHM).

Overall, these findings suggest that minority couples may have responded relatively more positively to the programs, on average, than other couples. That some positive effects fade is not a particularly unusual finding in studies of social interventions. I believe these findings may suggest an important, positive response to the interventions but also portray the need for something more and something continuing. An important question for the field lies in figuring out what those “somethings” look like in the lives of those interested in, and responsive to, such efforts.

A Related Finding on Ethnicity from Another Major Data Set

In one of our studies, we find an even more striking finding than what was found in BSF and SHM regarding impacts for ethnic minority couples. This particular study, funded by NIH,[xi] would be the single largest randomized trial in the history of the relationship education field if it were not for BSF and SHM. We have been evaluating the impact of a version of the intervention we have developed, refined, and tested over many years, called PREP[xii]  (the Prevention and Relationship Education Program). Adaptations of PREP were also used in some of the sites in the BSF and SHM studies.

My colleagues and I have worked with all branches of the military over the years. We have worked most closely with military Chaplains, who have a strong tradition of providing various relationship education services to military families. In our most recent paper from this project, we present analyses of impacts at two years following program delivery by U.S. Army chaplains. This paper is forthcoming in the same journal as Paul Amato’s paper mentioned above.[xiii] While we had found modest evidence of positive impacts on relationship quality post-intervention, two years later we found no evidence of sustained impacts on relationship quality. On the other hand, at the same two-year follow-up, we found that couples assigned to the intervention were significantly less likely to have divorced than couples in the control group.[xiv] This result has some parallels to the BSF results for Oklahoma, with some relationship quality impacts earlier on and a stability impact later on.[xv]

More to the purpose here, we found that minority couples received a far larger divorce reduction impact from the intervention than non-minority couples. Minority couples in the intervention group were about one-fourth as likely to divorce by the two-year point as minority couples in the control group. We also found a trend suggesting that couples who felt the most economic strain had larger divorce reduction impacts, and this economic strain effect was independent from the minority effect. Such positive impacts may well fade with longer-term follow-ups (or other positive impacts may emerge), but the existing findings at two years were striking in the degree to which minority couples received the greatest benefit in terms of divorce reduction. This, too, is good news, and it adds to the accumulating evidence that ethnic minority couples benefit at least as much, and sometimes more, from relationship education services as do other couples.

Research in this field marches on. Amidst the ongoing concerns and arguments, I believe there is some good news to consider as the field continues studying how to foster relationship stability and quality, both in general and specifically with those individuals and families who face great disadvantages. I believe it is good news that the Administration of Children and Families is moving systematically on a program of research to support increasing effectiveness in family-strengthening efforts.   

*

Disclosure:  I (along with many colleagues such as Howard Markman) have played a substantial role in the development of a variety of relationship education approaches that were used in some of the sites in the BSF and SHM studies, and that are also used in a number of the projects funded by the government around the U.S. I receive income from our company called PREP. Further, I have been a long-time adviser for the efforts in Oklahoma. Of greater weight for me is the fact that I do believe in trying to build on promising studies, practices, and program models in the areas I focus on here. You are entitled, of course, to disregard any of my viewpoints based on these facts, but I hope those with serious interest would grapple with the ideas and consider where they may have inherent merit.





[i] Johnson, M. D. (2012). Healthy Marriage Initiatives: On the need for empiricism in policy implementation. American Psychologist, 67(4), 296-308. (http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=2012-08242-001); See also: Hawkins, A. J., Stanley, S. M., Cowan, P. A., Fincham, F. D., Beach, S. R. H., Cowan, C., Rhoades, G. K., Markman, H. J., & Daire, A. P. (2013). A more optimistic perspective on government-supported marriage and relationship education programs for lower income couples: Response to Johnson (2012). American Psychologist, 68(2), 110-111. (http://psycnet.apa.org/journals/amp/68/2/110?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+apa-journals-amp+%28American+Psychologist%29)
[ii] Devaney, B., & Dion, R. (2010). 15-Month impacts of Oklahoma's Family Expectations Program. Washington DC: Mathematica Policy Research. (http://www.mathematica-mpr.com/publications/redirect_PubsDB.asp?strSite=PDFs/Family_support/BSF_15month_impacts.pdf)
[iii] This finding is in the final report for the BSF analyses at 36 months. P. 29 “At the three-year follow-up, 49 percent of BSF children in Oklahoma had lived with both parents continuously, compared with 41 percent of children in the control group (Table A.7b).” An 8% difference over 41% for control group is a 20% increase. Citation: Wood, R. G., Moore, Q., Clarkwest, A., Killewald, A., & Monahan, S. (2012). The long-term effects of Building Strong Families: A relationship skills education program for unmarried parents. Administration for Children and Families, Office of Planning, Research, and Evaluation.  Washington D. C. (http://www.acf.hhs.gov/sites/default/files/opre/bsf_36_mo_impact_report.pdf)
[iv] http://family-studies.org/can-we-strengthen-marriages-results-of-the-supporting-healthy-marriage-evaluation/
[v] For example: Wood, R. G., Moore, Q., Clarkwest, A., Killewald, A., & Monahan, S. (2012). The long-term effects of Building Strong Families: A relationship skills education program for unmarried parents. Administration for Children and Families, Office of Planning, Research, and Evaluation.  Washington D. C. (http://www.acf.hhs.gov/sites/default/files/opre/bsf_36_mo_impact_report.pdf); Lundquist, E. Hsueh, J., Lowenstein, A. E., Faucetta, K., Gubits, D., Michalopoulos, C., & Knox, V. (2014). A family-strengthening program for low-income families: Final impacts from the Supporting Healthy Marriage evaluation. OPRE Report 2014-09A. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. (http://www.acf.hhs.gov/programs/opre/resource/the-supporting-healthy-marriage-evaluation-a-family-strengthening-program-for-low-income-families-final-impacts-from-the)
[vi] Amato, Paul R. (2014). Does social and economic disadvantage moderate the effects of relationship education on couples? An analysis of data from the 15-month Building Strong Families evaluation. Family Relations, 63, 343-355. doi: 10.1111/fare.12069. (http://onlinelibrary.wiley.com/doi/10.1111/fare.12069/abstract)
[vii] Rauer, A. J., Adler-Baeder, F., Lucier-Greer, M., Skuban, E., Ketring, S. A., & Smith, T. (2014). Exploring Processes of Change in Couple Relationship Education: Predictors of Change in Relationship Quality. Journal of Family Psychology, 28(1), 65-76. (http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=2014-04342-001); Stanley, S. M., Amato, P. R., Johnson, C. A., & Markman, H. J. (2006). Premarital education, marital quality, and marital stability: Findings from a large, random household survey. Journal of Family Psychology, 20(1), 117-126. (http://psycnet.apa.org/index.cfm?fa=buy.optionToBuy&id=2006-03561-013)
[viii] Cowan, P., Cowan, C., Pruett, M., Pruett, K., & Wong, J. (2009). Promoting fathers' engagement with children: Preventive interventions for low-income families. Journal of Marriage & Family, 71(3), 663-679. (http://onlinelibrary.wiley.com/doi/10.1111/j.1741-3737.2009.00625.x/abstract)
[ix] See page xv: Wood, R. G., McConnell, S., Moore, Q., Clarkwest, A., & Hsueh, J. (2010). The Building Strong Families Project. Strengthening unmarried parents' relationships: The early impacts of Building Strong Families. Administration for Children and Families, Office of Planning, Research, and Evaluation. Washington D. C. (http://www.acf.hhs.gov/programs/opre/strengthen/build_fam/reports/unmarried_parents/15_impact_exec_summ.pdf)
[x] See page ES-7: Hsueh, J., Alderson, D. P., Lundquist, E., Michalopoulos, C., Gubits, D., Fein, D., & Knox, V. (2012). The Supporting Healthy Marriage Evaluation: Early Impacts on Low-Income Families.  Administration for Children and Families, Office of Planning, Research, and Evaluation.  Washington D. C. (http://www.acf.hhs.gov/sites/default/files/opre/early_impacts_low.pdf)
[xi] This project is supported by the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under Award Number R01HD048780. My comments here are solely my own responsibility and do not represent any official views of the National Institutes of Health.
[xii] The actual intervention manuals and materials are not available on the web but the general principles in PREP are most easily accessible in various books we have published, e.g.: Markman, H. J., Stanley, S. M., & Blumberg, S. L. (2010). Fighting for Your Marriage. San Francisco: Jossey-Bass.
[xiii] You can write to me to request a copy of the forthcoming paper if you wish. The citation is: Stanley, S. M., Rhoades, G. K., Loew, B. A., Allen, E. S., Carter, S., Osborne, L. J., Prentice, D., & Markman, H. J. (in press).  A randomized controlled trial of relationship education in the U.S. Army: 2-year outcomes. Family Relations.
[xiv] The divorce reduction impact held for the pooled analysis, but it was clearly driven by the larger of two sites—a site that was comprised of units much more involved in combat operations and high operational tempo, which also had younger married couples. The divorce reduction impact was non-existent for a smaller site where couples were older, more established, and not similarly as involved in major combat operations. Again, this is consistent with studies in the field where, when a difference emerges, couples at higher risk tend to get greater benefits from such services.
[xv] It is well recognized in this and other fields that one type of positive result can influence the odds of obtaining a different type of positive  result when one result (divorce) causes people to be missing for analysis of the other outcome (relationship quality). Researchers at Mathematica (the company that conducted the BSF evaluation) have written a paper on the depth of the challenges involved in resolving this dilemma in outcome studies. There is nothing approaching an ideal or perfect solution because data that are missing for meaningful reasons related to the goals of an intervention are simply not replaceable. See: McConnell, S., Stuart, E. A., & Devaney, B. (2008).  The Truncation-by-Death Problem: What to do in an experimental evaluation when the outcome is not always defined. Evaluation Review, 37(1), 157-186. (http://erx.sagepub.com/content/32/2/157.full.pdf