Impact Evaluation of the OBIRODH – Road to Tolerance Youth

DISCLAIMER This publication was produced for review by the United States Agency for International Development. It was

prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the

views of the United States Agency for International Development or the United States Government.

IMPACT EVALUATION OF THE

OBIRODH – ROAD TO TOLERANCE

YOUTH LEADERSHIP TRAINING

PROGRAM IN BANGLADESHI

UNIVERSITIES

FINAL REPORT

Prepared under Contract No. GS-10F-0033M / Order No. AID-OAA-M-13-00013, Tasking N058

DRG LEARNING, EVALUATION, AND

RESEARCH ACTIVITY

IMPACT EVALUATION OF THE OBIRODH –

ROAD TO TOLERANCE YOUTH

LEADERSHIP TRAINING PROGRAM IN

BANGLADESHI UNIVERSITIES

FINAL REPORT

July 2021

Prepared under Contract No. GS-10F-0033M / Order No. AID-0AA-M-13-00013, Tasking N058

Submitted to:

Mousumi Sarkar, COR

Submitted by:

Peter Vining, NORC Principal Investigator (New York University)

Cyrus Samii, NORC Principal Investigator (New York University)

Michael Gilligan, NORC Principal Investigator (New York University)

Contractor:

NORC at the University of Chicago

Attention: Matthew Parry, Program Manager

3450 East-West Highway, Suite 800

Bethesda, MD 20814

Telephone: (301) 634-5444

E-mail: Parry-[email protected]

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the United

States Agency for International Development or the United States Government.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | i

TABLE OF CONTENTS

ACRONYMS III

EXECUTIVE SUMMARY 1

1. INTRODUCTION 4

2. VIOLENT EXTREMISM IN CONTEMPORARY BANGLADESH 5

3. BACKGROUND ON PREVENTING AND COUNTERING VIOLENT EXTREMISM (P/CVE)

PROGRAMS 11

4. THEORY OF CHANGE 14

FACTORS INHIBITING THE IMPACT OF BYSTANDER INTERVENTION TRAINING 18

ADAPTING THE BYSTANDER INTERVENTION MODEL TO REDUCE VIOLENT EXTREMISM

ANTECEDENT BEHAVIORS 20

5. IMPACT EVALUATION METHODOLOGY 21

INTERVENTION DESIGN: ADAPTING BIT TO BANGLADESHI UNIVERSITIES 22

SITE SELECTION, PARTICIPANT RECRUITMENT, AND RANDOMIZATION PROCEDURE 24

SAMPLE SIZE, DATA COLLECTION STRATEGY, AND TIMELINE 26

OUTCOMES OF INTEREST AND MEASUREMENT STRATEGY 28

NORMS 29

SKILLS 32

CONFIDENCE AND SELF-ASSESSED COMPENTENCE TO ACT 33

WILLINGNESS TO INTERVENE 34

PEER LEADERS VS TRADITIONAL AUTHORITY FIGURES 35

BASELINE CHARACTERISTICS COVARIATE BALANCE 35

6. DATA ANALYSIS 38

7. RESULTS 41

NORMS 41

SKILLS 44

CONFIDENCE AND SELF-ASSESSED COMPETENCE TO ACT 45

WILLINGNESS TO INTERVENE 46

PEER LEADERS VS TRADITIONAL AUTHORITY FIGURES 48

PLACEBO OUTCOMES: DO NORMS AND CONFIDENCE TO ACT CHANGE WITH RESPECT

TO UNRELATED TOPICS 51

LIMITATIONS 52

SUMMARY 53

8. DISCUSSION AND CONCLUSIONS 54

APPENDIX A. 56

APPENDIX B. 63

TABLES

Table 1: Participating Public and Private Universities in Dhaka and Rajshahi 24

Table 2: Data Collection Timeline 28

Table 3: Pre-Treatment Covariate Balance 37

Table 4: Effects of Programming and Tolerance BIT Program on Norms of Tolerance 56

Table 5: Effects of Programming and Tolerance BIT Program on Religious Tolerance (Web-

Based Stimulus Activity) 56

Table 6: Effects of Programming and Tolerance BIT Program on Rejection of Violence 57

Table 7: Effects of Programming and Tolerance BIT Program on Internalized Skills 57

Table 8: Effects of Programming and Tolerance BIT Program on Confidence to Act 58

Table 9: Effects of Programming and Tolerance BIT Program on Willingness to Intervene 58

Table 10: Participant Identification with Peer Leaders, compared to Traditional Authority Figures,

among Those Enrolled in Tolerance BIT Program 59

Table 11: Effect of Peer Leaders, compared to Traditional Authority Figures, on Participant

Evaluation of Social Environment among Those Enrolled in Tolerance BIT Program 59

Table 12: Effect of Peer Leaders, compared to Traditional Authority Figures, on Participant

Identification with Program Goals among Those Enrolled in Tolerance BIT Program 59

Table 13: Impact of Peer Leaders, compared to Traditional Authority Figures, on Outcomes of

Interest among Those Enrolled in Tolerance BIT Program 60

Table 14: Summary Statistics for Outcomes of Interest 60

Table 15: Standardized Baseline Covariate Balance by Attrition 62

Table 16: Gross Effect of Programming and Effect of Tolerance BIT Curriculum on Placebo

Norms and Confidence Outcomes. 62

FIGURES

Figure 1: Pathways from BIT Programming to Willingness to Intervene 17

Figure 2: Effects of Programming and Tolerance BIT Curriculum on Norms of Tolerance 42

Figure 3: Effects of Programming and Tolerance BIT Curriculum on Norms of Religious Group

Status Measures 42

Figure 4: Effects of Programming and Tolerance BIT Curriculum on Rejection of Violence and

VEOs 43

Figure 5: Effects of Tolerance BIT Curriculum on Common Knowledge Assessment Scores 44

Figure 6: Effects of Programming and Tolerance BIT Curriculum on Internalized BIT Skills 45

Figure 7: Effects of Programming and Tolerance BIT Curriculum on Confidence Act 46

Figure 8: Effects of Programming and Tolerance BIT Curriculum on Willingness to Intervene 47

Figure 9: Participant Identification with Peer Leaders, compared to Traditional Authority

Figures, among Those Enrolled in Tolerance BIT Program 48

Figure 10: Participant Identification with Social Environment of Programming, comparing Peer

Leaders to Traditional Authorities among Those Enrolled in Tolerance BIT Program 49

Figure 11: Participant Identification with Programming Goals, Comparing Peer Leaders to

Traditional Authorities, among Those Enrolled in Tolerance BIT Program 50

Figure 12: Impact of Peer Leaders, compared to Traditional Authority Figures, on Outcomes of

Interest among Those Enrolled in Tolerance BIT Program 51

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| ii

ACRONYMS

BIT Bystander Intervention Training

CVE Countering Violent Extremism

DID Difference in Differences

DIM Difference in Means

GTD Global Terrorism Database

IE Impact Evaluation

IPA Innovations for Poverty Action

ITT Intent-to-Treat

LGAP Local Government Accountability and Performance

LGBT Lesbian Gay Bisexual Transgender

NPC No-program Control

P/CVE Preventing and Countering Violent Extremism

RCT Randomized Controlled Trial

SMS Short Message Service (text messaging using standard communication protocols)

UNICEF United Nations Children’s Fund

USAID United States Agency for International Development

USD U.S. Dollar

VEO Violent Extremist Organizations

WGI Worldwide Governance Indicators

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| iii

EXECUTIVE SUMMARY

This report presents the results of a rigorous randomized impact evaluation of a youth leadership

training program aimed at preventing violent extremism in Bangladesh. The program was part of the

broader USAID-sponsored Obirodh: Road to Tolerance project in Bangladesh, which aimed to support

Bangladeshi civil society actors in their work to promote tolerance and mitigate the spread of

extremism. The USAID Center for Democracy, Human Rights, and Governance (DRG) worked with the

USAID Bangladesh mission to commission an evaluation as part of the DRG Learning, Evaluation, and

Research (DRG-LER) Activity. The key finding from the evaluation was that the program significantly

improved participants’ norms of tolerance and their expected willingness and capacity to intervene safely

and effectively when they witness intolerant and extremist behavior. Thus, the program represents a

promising approach to addressing violent extremism that could be applied in a variety of contexts.

PROGRAM DESCRIPTION

Preventing and countering violent extremism (P/CVE) suffers from a needle-in-a-haystack problem:

becoming a violent extremist is a very low probability event. Therefore, targeting programs on at-risk

persons wastes resources on de-radicalization programs for people who never would have been

radicalized in the first place. In addition, strategies attempting to deter or use surveillance on entire “at-

risk” groups in an aggressive and coarse manner may do more to alienate members of that group,

undermining the potential for enlisting the support of those who may actually have no interest in violent

extremism. Moreover, the context in which a P/CVE program is being implemented may greatly

constrain the menu of feasible strategies. Approaches taken from within industrialized liberal

democracies that rely on the police, justice system, or youth service institutions may not be appropriate

in contexts where civil liberties are constrained or police forces are partisan or abusive. The program

examined by this evaluation sought to avoid these pitfalls by focusing instead on making the social

climates in which violent extremists might recruit less hospitable to violent extremism. It did so by

training youth to recognize expressions of intolerant and extremist views and to intervene safely and

effectively to counter such expression. Specifically, rather than attempting to identify and neutralize

individual extremists, this program attempts to enable those who do not espouse intolerant or

extremist views to contribute to a general climate in which expressions of intolerance or violence

would be countered. By training individuals to publicize and reinforce norms against extremist views and

behaviors, the programming is meant to both decrease the likelihood of extremist expression among

those holding attitudes that might lead them to express it, and empower others who hold attitudes

opposed to extremist behaviors to further reinforce norms against it. The program thus offers a new

approach to P/CVE not relying on surveillance or aggressive police action, and an approach to which

broader segments of the population falsely assumed to be “at-risk” may be more receptive and

responsive.

The youth leadership training program focused on university students because the majority of those

involved in recent violent attacks in Bangladesh have been university graduates with educated, middle-

class origins. We selected six universities as sites for the study, three in Dhaka and three in Rajshahi.

The program drew inspiration from “bystander intervention training” (BIT) programs implemented in

other contexts to address anti-social behaviors. The program trained university students to identify and

intervene in situations in which their peers expressed extreme intolerance or aggression toward

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 1

vulnerable groups or minorities, with a particular focus on religion and gender. The impact evaluation

used a randomized controlled trial design, testing the pro-tolerance BIT curriculum against a placebo

curriculum and against a no-program control group. This allows the evaluation to estimate both the

gross effect of the program, which involves both novel social interactions and generic prompts to think

critically along with the content of the curriculum, and the effect of the BIT curriculum itself. The

evaluation also examined whether it would be more effective to have peer leaders (those similar to the

trainees) rather than traditional authority figures (professors, in this case) deliver the training. The study

used surveys and a web-based stimulus activity to measure attitudes, the internalization/retention of

knowledge and skills, and behavioral intentions.

FINDINGS

The program led to improved norms of tolerance among participants, although much of this

improvement appears attributable to the nature of the social interactions created by the program and

the generic prompts to get subjects to think critically rather than the BIT curriculum itself. It is plausible,

for example, reduced sexism observed among participants is attributable to the mixed-gender nature of

the programming, rather than the BIT curriculum content. We can distinguish these sources of the

effects when we compare how BIT program participants fared relative to those who received no

program at all and relative to those who received a placebo curriculum (that focused on health and

sanitation). The BIT program successfully improved participants’ skills and competence to intervene in a

potentially effective way when presented with scenarios of aggression, with similar caveats suggesting

these effects are partially driven by social interactions within the program. The BIT program also

improved participants’ eagerness and confidence to act when they witnessed intolerance, and these

effects appear mainly to be derived from the content of the BIT curriculum itself. Finally, we find the BIT

program increased participants’ willingness to intervene (the primary outcome of interest, as measured

by behavioral intentions in response to vignettes) in a variety of perpetrator-victim scenarios.

Importantly, the curriculum was able to increase tolerant attitudes not only towards groups directly

addressed by the curriculum (women, non-Muslims), but also in an area our implementing partner

deemed too controversial to include, namely LGBT rights. Students appear to have extended lessons

learned on tolerance to this latter group even though the curriculum did not directly address it.

Moreover, the program increased willingness to intervene against violent extremism, even though the

curriculum was focused mostly on positive reinforcement of tolerance and anti-violence norms.

With regard to peer versus traditional authority facilitators, the evidence suggests on the whole, the

traditional authority facilitators were more effective. These findings suggest a “social climate” strategy,

i.e., a strategy designed to empower and encourage those already holding tolerant attitudes to safely

protect and promote norms of tolerance within their social milieus, may be potentially effective for

P/CVE purposes, especially in contexts with heightened political sensitivity and frequent rights violations.

While encouraging, these findings do not shed light on the longer-term impact of such programming, or

whether changes within the social microclimates of participants have endured following, for example, the

disruptions of the Covid-19 pandemic. In addition, while this program was designed as a “megadose”

(about 28 hours) of BIT when compared to similar initiatives, it is unclear to what extent scaling the

programming (either through larger class sizes and/or shorter duration) would potentially erode efficacy.

Finally, although the research design sought to minimize demand effects and social desirability bias in

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 2

survey responses and activities, it is nevertheless possible some of the attitudinal measures we use

reflect these issues.

IMPLICATIONS

The implications of our results are that indirect, “social climate” strategies are a compelling approach to

address violent extremism. While further study should be conducted on the durability of these

interventions before strong policy conclusions are drawn, the BIT curriculum implemented in

Bangladesh was successful and should be considered for other contexts. To that end, this evaluation’s

findings suggest some lessons learned for future P/CVE endeavors. In future programming, relying on

peer facilitators appears to be somewhat risky because peer facilitators appeared to be less adept

instructors. Thus, we recommend relying on professional instructors for future training programs like

the one studied here. Second, this evaluation suggests that a BIT approach focusing on empowering

those holding tolerant attitudes to safely protect and promote tolerant social norms can cultivate

tolerant attitudes toward groups too politically or socially sensitive and controversial to address

directly.

The program in this study was designed to make the environment less hospitable to violent extremists

by training members of civil society to recognize intolerant and extremist views and take appropriate

action. The program was successful in meeting these goals, producing significant improvements in

participants’ norms of tolerance and their willingness and ability to intervene safely and effectively when

they witness intolerance and extremism. These improvements were reflected in survey instruments

used to measure attitudinal changes, skills/knowledge retention and application using aptitude measures,

and behavioral intentions in response to vignettes.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 3

1. INTRODUCTION

As part of the DRG Learning, Evaluation, and Research (DRG-LER) Activity, USAID requested that

NORC design and implement an impact evaluation (IE) to study a youth leadership training program

aimed at preventing violent extremism in Bangladesh. The program was part of the broader USAID-

sponsored Obirodh: Road to Tolerance project in Bangladesh, which aimed to support Bangladeshi civil

society actors in their work to promote tolerance and mitigate the spread of extremism. The youth

leadership training program focused on university students, given that the majority of those involved in

recent violent attacks in Bangladesh have been university graduates with educated, middle class origins.

Countering violent extremism is fraught with the reality that such extremism is, by its nature, rare, and

strategies that attempt to deter or use surveillance on presumed “at-risk” communities in an aggressive

and coarse manner may do more to alienate those who have no interest in violent extremism than

identify or temper those who are genuinely at-risk. Such programs, like the New York Police

Department’s “Muslim Surveillance and Mapping Program” were found to be both illegal in profiling faith

communities and also stifling of potential cooperation from community members. We discuss examples

of such approaches and their negative consequences in an overview of P/CVE strategies found in section

3 of this report.

Working with university students in Bangladesh, the youth leadership training program sought to avoid

such pitfalls by using positive reinforcement of tolerance and anti-violence norms to cultivate a social

“climate” against extremism. The program drew inspiration from “bystander intervention training” (BIT)

programs implemented in other contexts, such as universities, secondary schools, and organized sports,

to address anti-social behaviors. It sought to train university students to identify and intervene, in ways

safe for themselves and others, in situations in which their peers expressed extreme intolerance or

aggression toward vulnerable groups or minorities, with a particular focus on religion and gender. The IE

used a randomized controlled trial (RCT) design, testing the pro-tolerance BIT curriculum against a

placebo curriculum and against a no-program control group. The placebo was designed as a parallel

leadership development program meant to build critical thinking skills with a focus on civic issues

unrelated to tolerance. Using a placebo allowed the evaluation to estimate both the gross effect of the

program, which involves both novel social interactions in a programmatic context and generic prompts

to think critically, and the valued added by the content of the curriculum per se. The evaluation also

studied whether it would be more effective to have peer leaders (those similar to the trainees) leading

sections of the programming as “facilitators,” rather than traditional authority figures (i.e., professors in

this case).

This report presents the results of the IE. We find that the BIT program led to improved norms of

tolerance among participants, although much of this improvement appears attributable to the nature of

the social interactions and generic prompts to think critically that were created by the program. We

also find that the BIT program successfully improved participants’ skills and judgement (competence)

when presented with scenarios of aggression, with similar caveats suggesting that these effects are

partially driven by the general social interactions within the program. Furthermore, we find that the BIT

program improved eagerness and confidence to act among participants, which we attribute mainly to the

focused curriculum of the BIT program. Finally, we find that the BIT program led to willingness to

intervene (the primary outcome of interest) when participants were presented with a variety of

perpetrator-victim scenarios. Presumably, willingness to intervene is driven by the effect of confidence

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 4

and self-assessed competence to act among BIT participants. The results are promising for indirect,

social climate strategies to counter violent extremism.

With respect to the aforementioned outcomes (norms, competence, confidence, and willingness to

intervene), we found suggestive evidence that peer leaders may be less effective than traditional

authority figures (namely, professors) at leading a program of this nature, although these differences

were not large. Differences between each type of facilitator appear more pronounced regarding

participant perceptions of the facilitator leading their groups, which may be driving small differences in

the other outcomes of interest. Not surprisingly, traditional authority figures (i.e., professors) were

significantly more likely to be seen as experienced, while peer leaders were seen as youthful.

Interestingly, the traditional authority figures were perceived as more approachable than the peer

leaders. These results speak to the importance of experience and professional skill in administering

training programs.

In section 2 of this report (below), we begin with a discussion of the context, describing the nature of

violent extremism in contemporary Bangladesh. Violent extremism in Bangladesh regularly involves

educated young adults, which motivates our focus on university students. In section 3, we review

current strategies for countering violent extremism, explaining why we think the bystander intervention

training approach is most suitable for university students in the present context. In section 4, we present

the theory of change for the youth leadership training program. Section 5 describes the research design

and data collection plan for the IE. Section 6 then present our statistical analysis of the results of the

evaluation. A concluding section 7 summarizes what we have learned, derives policy implications, and

offers suggestions for further research.

2. VIOLENT EXTREMISM IN CONTEMPORARY BANGLADESH

Violent extremism in Bangladesh has increased noticeably in recent years, threatening development goals

and political stability in the world's eighth most populous country. The Institute for Economics and

Peace’s Global Terrorism Index records an increase on its 10 point “Impact of Terrorism" scale from 4.1

to 5.9 between 2012 and 2015 in Bangladesh.

Estimates of the number and severity of extremist attacks

in recent years vary, depending on the data source and criteria used in defining such attacks. One

conservative estimate claims that the 2013-2015 period was marred by at least 47 major extremist

attacks, killing 54 and injuring hundreds more, in contrast to just 92 attacks during the entire decade

prior

. The number of known violent extremist organizations (VEOs) also varies widely, as groups often

merge, splinter, rebrand, or operate in conjunction with non-violent political wings; estimates have

ranged from 12 to 70 organizations.

An independent analysis of data provided by the Global Terrorism Database (GTD)

, which records

deliberate acts of violence for political, religious, economic or social purposes, suggests that five major

VEOs have carried out 114 attacks, killing at least 127 people and injuring another 348 people within

Riaz, A. (2016a). Who are the Bangladeshi ‘Islamist Militants? Perspectives on Terrorism, 10 (1).

Rahman, M. A. (2016). The Forms and Ecologies of Islamist militancy and terrorism in Bangladesh. Journal for Deradicalization,

(7), 68-106.

Ahsan, Z. (2005). Trained in foreign lands: The spread inland. The Daily Star, 21 August 2005, and Rahman, M. A. & Kashem,

M. B. (2011). Understanding religious militancy and terrorism in Bangladesh. Dhaka: ICA Bangladesh.

National Consortium for the Study of Terrorism and Responses to Terrorism (Start). (2017). Global Terrorism Database

[Data File]. Retrieved from https://www.start.umd.sdu/gtd.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 5

Bangladesh from 2013-2016. These include VEOs with an international presence (Islamic State in

Bangladesh and Al Qaeda in the Indian Subcontinent, known within Bangladesh as Ansar-al-Islam), as well

as Bangladeshi organizations (Ansarullah Bangla Team, Jamaat-E-Islami Bangladesh, and Jama'atul

Mujahideen Bangladesh). These organizations are usually responsible for the most high-profile and mass-

casualty events, such as the July, 2016 Holey Cafe attack and hostage incident, which killed 28 people

and wounded another 30. This attack was claimed by Islamic State, and likely also involved Jama'atul

Mujahideen Bangladesh.

Many VEO-sponsored attacks in Bangladesh have targeted individuals rather than institutions, usually for

sectarian purposes, and as part of an ongoing campaign to intimidate and terrorize different minority

groups within the country. Targets have included ethnic, religious and sexual minorities, as well as

foreigners, academics, journalists, and those deemed to have committed acts of blasphemy or apostasy

against extremist interpretations of Sunni Islam. GTD data indicate that of the 59 lethal VEO-related

attacks recorded from 2013-2016, at least 35 were murders of targeted individuals; victims have ranged

from Hindu, Shia and Christian religious leaders

to university professors,

atheist and secular bloggers,

lesbian, gay, bisexual, and transgender (LGBT) rights activists,

foreign aid workers,

and others. The

April, 2016 murder of LGBT activist and US Embassy employee Xulhaz Mannan along with colleague

Mahbub Rabbi Tonoy by Ansar-al-Islam is a prototypical example of these targeted killings, which are

often preceded by death threats and carried out by groups of men wielding knives, machetes, or other

crude weapons.

In addition to these high-profile attacks carried out by or on behalf of known extremist organizations,

Bangladesh has also experienced a troubling wave of sectarian violence carried out by individuals or

unorganized small groups against the aforementioned minority groups. The Global Terrorism Database

contains information on many attacks whose perpetrators remain unidentified, but clearly fall within this

narrative. Oftentimes, these attacks include difficult-to-prosecute property crimes (such as arson) and

are carried out using machetes, knives, rocks, petrol bombs and crude incendiary devices. Examples

from the 2013-2016 period include attacks against Sufi Muslims

, Hindu temples

and homes

, as well

as religious minority leaders.

Profiles of the individual militants themselves contradict a common narrative that they tend to be

uneducated and unskilled. Many arrestees from recent plots and attacks have been university graduates

with educated, middle class origins, suggesting that economic deprivation offers limited explanatory

power for why individuals join VEOs.

An analysis of the socio-demographic profiles of Bangladeshi

"IS 'beheads' Hindu priest in Bangladesh," BBC, February 21, 2016; "Top Shia Preacher Killed in IS Claimed Attack in

Bangladesh," Outlook India, March 15, 2016; "Bangladesh: Another Hindu priest murdered," The Daily Star Online, July 1, 2016;

"B'desh Christian priest attacked in his house by armed men," Deccan Herald, October 6, 2015.

"Professor murder: Militants claim responsibility on Facebook," Dhaka Tribune, November 16, 2014.

"Ananta Bijoy Das Hacked To Death In Bangladesh In Third Such Killing Of Atheist Bloggers," International Business Times,

May 12, 2015; "Second blogger hacked to death this year in Bangladesh," Reuters, March 30, 2015.

"Islamist Militants Suspected in Killing of Gay Rights Activist in Bangladesh," The New York Times, April 26, 2016.

"Australia: Bangladesh Opposition officials among seven charged over Italian aid worker murder," ABC Online, June 28, 2016.

"3 Sufi Muslims attacked in B’desh," New Delhi Pioneer, July 30, 2016.

"Dhaka: Fresh Attacks on Hindu Temples Create 'Widespread' Panic Among Minorities," The Daily Star Online, March 9,

2013.

"Bangladesh: 10 Hindu houses torched in Dinajpur," Dhaka Tribune Online, December 4, 2016.

"Leader of religious minority forum attacked in Bangladesh," Deutsche Presse-Agentur, November 24, 2015.

Rahman, M. A. (2016). The Forms and Ecologies of Islamist militancy and terrorism in Bangladesh. Journal for Deradicalization,

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 6

militants arrested from January 2014 through June 2015 is consistent with this claim; the majority of

them are young, well-educated men from middle class backgrounds. Out of 112 alleged militants

arrested during this time period, 52 were men between the ages of 18 and 30. Out of 65 for whom

occupations could be identified, 53 held professional or middle-class jobs/career tracks. Nine were

university students, nine were engineers, five were teachers, and the rest were a variety of other

middle-class professions including business owners, travel agents, IT experts, and office clerks. In

contrast, only 13 of the 65 held jobs in working class or manual labor positions. Regarding education,

nearly all of the 53 individuals in middle class career tracks hold or are pursuing college degrees, and

several hold advanced degrees. Only 13 were madrassa students or teachers.

Evidence from student and school administrator focus groups can provide some insight into possible

explanations for the aforementioned patterns in VEO perpetrators and targets. Focus groups of

university and post-secondary level madrassa students held by Breakthrough Media found that

intolerance was fostered by the Bangladeshi secondary education system, due to divisions that the four

different types of secondary schools (English, Bangla, Quomi madrassa and Aliya madrassa) create and

exacerbate along lines of gender, wealth, religion, and religiosity, without building a sense of empathy for

different backgrounds. In addition, all four school types were found to promote a religiously/culturally

narrow definition of Bangladeshi national identity that is closely tied to Sunni Islam. These findings from

student focus groups were corroborated by a survey of school administrators in which 72% responded

that the “diverse streams of education [i.e., the four different school types] contribute to radicalization

in Bangladesh.”

Apparently, despite universities being more integrated than the secondary schools, the

economic and cultural divisions originating in the primary and secondary education systems persist

within the university system while the financial and social vulnerabilities of university students – who are

typically facing academic pressures, and the pressures of early adulthood and identity formation in the

new and unfamiliar settings of a university campus, potentially with few preexisting social connections-

also make them more susceptible to outside influences and pressures.

Country-wide survey evidence from Bangladesh also suggests a concerning political and social

environment. Although three-quarters of respondents in a 2012 country-wide survey of Bangladeshi

Sunni Muslims felt that suicide bombings and other attacks against civilian targets were rarely or never

justified, roughly a quarter responded that they were sometimes or often justified; these levels of

support for suicide attacks were also found to be substantially higher than those within the broader

region (including Pakistan, Malaysia and Indonesia).

Further analysis of the survey reveals a population that appears to be split between strict and moderate

cultural interpretations of Sunni Islam with respect to a variety of issues. While a majority of

respondents stated that women should have the right to divorce their husbands and choose whether to

wear the hijab, sizable minorities (about 30%) responded that they should not; moreover, a sizable

minority (35%) responded that honor killings of women by family members are often or sometimes

justified. Regarding attitudes towards minority groups, Sunni Muslims in Bangladesh tend to be

(7), 68-106.

Riaz, A. (2016). Who are the Bangladeshi ‘Islamist Militants? Perspectives on Terrorism, 10(1).

Breakthrough Media. (2017). “Research Paper for Credible Voices Bangladesh.” Report prepared for USAID.

Ibid.

Fair, C. Christine, Ali Hamza, and Rebecca Heller. “Who Supports Islamist Militancy in Bangladesh: What the Data Say?”

Available at SSRN 2804275 (2016).

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 7

conservative. About 70% responded that homosexual behavior was “morally wrong,” 68% responded

that they would be “not too” or “not at all” comfortable if a son of theirs someday married a Christian,

and about 16% did not consider Shia to be Muslims. Regarding attitudes towards the West, only about

17% responded that Western media have hurt morality within Bangladesh.

Regarding attitudes towards

their fellow Muslims, majorities or near-majorities favored the stoning of people who commit adultery

(52%), punishments like whipping or cutting off of hands for the crimes of theft or robbery (50%), and

the death penalty for people who leave the Muslim religion (44%).

A more recent set of surveys conducted between 2012-2014 commissioned by PACOM and carried out

by Nielson-Bangladesh roughly corroborates many of the findings in the 2012 Pew study regarding

perceptions on violence and foreign countries; surveys in 2012, 2013, and 2014, all found that roughly

half of respondents agreed that the use of violence was acceptable “against immoral people,” and “to

maintain the culture and traditions of society,” while between 80-90% agreed that violence was

acceptable “in defense of one’s religion.” Regarding perceptions of Western countries, less than half of

respondents reported positive perceptions of the United States’ influence on Bangladeshis’ way of life,

while less than a quarter reported positive perceptions of the United Kingdom.

Finally, recent work has also suggested that many of the aforementioned patterns both in the individuals

recruited/targeted by VEOs, as well as the prevailing social/political context, are in part exacerbated by

the proliferation of social media use (especially Facebook) in an environment that is still characterized by

significant levels of illiteracy and a lack of critical thinking education.

Both a SecDev study on behalf of

UNDP Bangladesh and the Breakthrough Media focus groups reported that Facebook (which about 80%

of the population use) has contributed both to increasing “cultural tribalism” via information echo

chambers, while also offering a tool for VEO messaging towards vulnerable populations, despite findings

that very few Bangladeshis actually seek out extremist pages and content. Interestingly, Madrassa student

focus groups reported that they viewed news shared on Facebook as more trustworthy than “the

media,” due to their ability to self-select their content.

Extremist groups make use of this cultural

tribalism in developing graphical/video content designed to provoke emotional reactions and reinforce a

sense of victimhood among Sunni Muslims at the hands of purportedly hostile minority groups and

foreign powers, often by connecting them to injustices and conflict narratives abroad.

The aforementioned context and profiles of those carrying out recent extremist attacks suggest that

economic “countering violent extremism” (CVE) interventions (i.e., jobs programs, cash transfers, etc.),

which have been attempted in numerous other contexts with frequently unconvincing results, are likely

ill-suited as effective strategies for the Bangladeshi setting.

Given that most recent perpetrators have

The Pew Research Center. (2012). The World’s Muslims [Data File]. Retrieved from http://www.pewforum.org/datasets/2012/.

Ibid.

“Vulnerability to Extremist Influence in Bangladesh.” (June, 2012). Report commissioned for the United States Pacific

Command (PACOM).

The SecDev Group. (2016). “Violent Extremist Narratives and Social Media in Bangladesh.” Report commissioned by the

UNDP Bangladesh.

“Vulnerability to Extremist Influence in Bangladesh” p. 7.

“Violent Extremist Narratives and Social Media in Bangladesh.” (p. 7); “Bangladesh Country Needs Assessment: Drivers of

Radicalization and Recruitment, and Community Resilience to Violent Extremism.” (March, 2016). Assessment conducted for

the Global Community Engagement and Resilience Fund by the Royal United Services Institute.

For a more thorough discussion of economic CVE interventions and their merits, see Gilligan, M., Samii C., and Vining, P.

(2017). Evidence Review Paper: Countering Violent Extremism. Paper prepared for USAID Impact Evaluation Clinic, March 27-

31, 2017.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 8

not only been gainfully employed, but are often educated members of the middle class, it is far more

likely that CVE efforts need to incorporate a psychosocial understanding of radicalization and prevention

efforts. Specifically, such efforts should be premised on the understanding that individuals who are

successfully targeted for recruitment by VEOs often have the same prosocial dispositions as those who

become involved in other social movements and organizations; by becoming involved in VEOs, they are

attempting to fulfill their psychosocial needs and desires in ways that traditional kinship, peer and

community ties are apparently unable to. While involvement in VEOs carries potentially high costs to

those individuals, the payoffs of fulfilling a “quest for significance” can outweigh those costs, especially if

the broader political/social context reduces barriers to entry and the risks of punishment.

It is well known that individuals are more willing to sacrifice their own welfare in order to pursue

feelings of efficacy in service of a cause, oftentimes in response to events that lead to feelings of

“significance loss."

Such significance loss events may include sources of trauma that cause feelings of

loss, shame, humiliation, deprivation, disenfranchisement, vulnerability, reduced self-worth,

isolation/loneliness, conflicted identity, and/or disillusionment with one’s current social role.

Such

feelings lead to cognitive openings within the individual, i.e., an openness to reorientation of their beliefs.

In order for an individual’s quest for renewed significance to manifest as participation in violent

extremism, however, these psychological motivations require (1) “moral warrants" that justify the use of

violence, as well as (2) social processes of introduction and involvement to exploit the identified

cognitive openings. VEOs thus step in to fulfill these necessary conditions by providing moral/ideological

warrants for violence in socially empathetic settings, in addition to identifying, grooming, and at times

even creating vulnerable individuals.

Crucially, social affirmation is understood to be a big part of how

participation in violent extremism fulfills the need for significance. The type of climate strategy studied in

the current IE attempts to create a counterweight to social affirmation through extremism.

Introduction to VEOs often occurs via kinship and friendship networks, although some perpetrators

were found to have “self-radicalized.” Introduction is usually gradual, starting with passive participation

in informational events or low-risk activism activities. Assuming that the VEO is interested in more

involvement from a potential recruit, further indoctrination occurs via radicalized elites, usually in the

context of a new and supportive community. This often occurs concurrently with isolation from and

rejection of moderating influences and old social ties. As a part of this socialization process, sacrificial

behavior may be required as a signal of reliability for future group contributions.

A wide variety of programs and initiatives have attempted to disengage and deradicalize individuals from

VEOs. Almost exclusively, these programs have targeted those who have either already committed, or

were at imminent risk of committing an act of violence. The choice of this target population is largely

due to the difficulties of detecting and targeting interventions at those in earlier stages of radicalization.

Most deradicalization programs have attempted to leverage some combination of implicit interventions

Ibid.

Dugas, M., Bélanger, J. J., Moyano, M., Schumpe, B. M., Kruglanski, A. W., Gelfand, M. J., … & Nociti, N. (2016). The quest for

significance motivates self-sacrifice. Motivation Science, 2(1), 15.

Bushman, B. J. (Ed.). (2016). Aggression and Violence: A Social Psychological Perspective. Psychology Press; Agnew, R. (2010). A

general strain theory of terrorism. Theoretical Criminology, 14(2), 131-153.

Kruglanski, A. W., Gelfand, M. J., Bélanger, J. J., Sheveland, A., Hetiarachchi, M., & Gunaratna, R. (2014). The psychology of

radicalization and deradicalization: How significance quest impacts violent extremism. Political Psychology, 35(S1), 69-93.

Berman, E. (2011). Radical, religious, and violent: The new economics of terrorism. MIT press; Wiktorowicz, Q. (2005).

Radical Islam rising: Muslim extremism in the West. Rowman & Littlefield Publishers.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 9

(i.e., those addressing the underlying reasons for joining) and explicit interventions (i.e., those countering

the radicalized worldview).

The efficacy of these programs is difficult to evaluate, due to: (1) a lack of

RCTs used to evaluate their effects, (2) strong demand effects leading participants to report being

“cured” (which often leads to leniency, reduced sentencing, and/or other incentives), and (3) because

post-program monitoring makes participants less-appealing for risk-averse VEOs to approach for

recidivism.

One example of a series of interventions designed to address radicalization at earlier stages was the

German HAYAT program, which seemed to offer a fruitful model for preventing highly radicalized (but

not yet violent) individuals from emigrating to fight on behalf of jihadist groups in Syria/Iraq during the

latter 2010’s. The program featured an innovative, community-based approach that sought to leverage

friend and family networks in order to deter a radicalized individual from departing.

Still, the success of

an individually-focused model for those at earlier stages of radicalization presumes a social and political

context in which such attitudes and behaviors noticeably depart from prevailing social norms, are

detectable by the individual’s social networks, and are therefore likely to put the individual at risk of

ostracization and/or harm should they become further involved in VEO activity. Moreover, the success

of such a model presumes that safe “off-ramps” are available for the individual’s close network to pursue

in order to intervene without risking legal consequences or ostracization for the radicalized individual or

themselves. In contexts where individual rights are regularly violated, these conditions may not be in

place.

Overall, while the psychosocial perspective and potential interventions derived from it offers a nuanced

post-hoc explanation for why individuals are targeted by, join, and carry out acts of violence on behalf of

VEO’s, it offers frustratingly few prescriptions for effective and targeted approaches to prevent or

counter violent extremism at the individual level, and at stages of involvement that precede imminent

plots and acts. Most of the aforementioned findings are based on studies of individuals who have either

been arrested/imprisoned following VEO-related offenses,

or who have voluntarily disengaged/de-

radicalized and agreed to share their accounts with researchers. Other findings are based on voluntary

accounts given by radicalized, but explicitly non-violent individuals in contexts where such organization

and activism is legal as protected speech.

Finally, because the population of potentially at-risk individuals is enormous while the population of

those who actually radicalize, join, and go on to commit acts of violence is so small, it is likely more

useful to target early-stage preventing and countering violent extremism (P/CVE) approaches at the

community level, rather than at the individual level. Specifically, rather than attempting to identify and

treat “at-risk individuals,” strategies that attempt to address social norms regarding behaviors and

Kruglanski et al. (2014); Refer to Gilligan, Samii and Vining (2017) for examples.

Koehler, D. (2013). Family Counselling as Prevention and Intervention Tool Against ‘Foreign Fighters’. The German ‘Hayat’

Program. Journal Exit-Deutschland. Zeitschrift für Deradikalisierung und demokratische Kultur, 3, 182-204.

See, for example, Kruglanski, A. W., Gelfand, M. J., Bélanger, J. J., Sheveland, A., Hetiarachchi, M., & Gunaratna, R. (2014). The

psychology of radicalization and deradicalization: How significance quest impacts violent extremism. Political Psychology, 35(S1),

69-93; Bélanger, J. J., Caouette, J., Sharvit, K., & Dugas, M. (2014). The psychology of martyrdom: making the ultimate sacrifice

in the name of a cause. Journal of Personality and Social Psychology, 107(3), 494.

See, for examples, Wiktorowicz, Q. (2005). Radical Islam rising: Muslim extremism in the West. Rowman & Littlefield Publishers;

Kenney, M., Horgan, J., Horne, C., Vining, P., Carley, K. M., Bigrigg, M. W., ... & Braddock, K. (2013). Organisational adaptation

in an activist network: Social networks, leadership, and change in al-Muhajiroun. Applied Ergonomics, 44(5), 739-747.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 10

attitudes towards groups targeted by VEOs are likely to offer an approach to P/CVE to which those “at-

risk” and their close-networks may be more receptive and responsive.

3. BACKGROUND ON PREVENTING AND COUNTERING

VIOLENT EXTREMISM (P/CVE) PROGRAMS

A variety of P/CVE programs have been implemented by governments and communities throughout the

post-9/11 period to date.

These programs can be loosely defined as those designed to prevent or

mitigate violent extremism by using a range of soft policy interventions

that target individuals, groups,

communities and/or societies. While many of these initiatives have been successfully implemented from

a programmatic standpoint,

very few of them have been implemented in such a way that enables

policymakers to determine whether they actually work- i.e., whether they achieve their goal of

preventing and/or countering radicalization and involvement in VEOs. Among the few P/CVE programs

that have included an evaluation component, virtually none have been designed in such a way that

enables causal inference from program effects. A 2017 systematic review of 73 CVE studies, for

example, found that only 14 were “effect evaluations” (i.e., those addressing causal effects of the

program), of which only 5 were quantitative IEs, and only 2 of which were quasi-experimental.

To our

knowledge, no P/CVE programs have yet been implemented as RCTs, which are the gold standard for

evaluating program effects.

The challenges to rigorous evaluation of P/CVE program effects are numerous, but fairly well-known.

First among these challenges is correctly reaching “at-risk” individuals or groups, without generating

unacceptable levels of false positives and/or exacerbating the problem by alienating or stigmatizing them.

Attempts to correctly profile “at-risk” individuals have largely proven fruitless, and have sometimes

backfired in the form of racial or religious profiling. The United Kingdom’s “Prevent” program, for

example, was largely blamed for resulting in the stigmatization of UK Muslims who expressed

conservative viewpoints; the mere inclusion of such individuals in programming designated as

“preventing violent extremism” made individuals feel targeted as “potential terrorists.”

Likewise, “at-

risk community” strategies risk similar blowback, especially when targeted at faith communities. Such

programming, like the New York Police Department’s “Muslim Surveillance and Mapping Program,” can

result in discriminatory profiling that ultimately alienates members of the communities they are

ostensibly intended to engage cooperatively and who are best-positioned to detect and potentially

intervene in actual cases of violent extremism.

For a helpful and comprehensive review of CVE program studies prior to 2017, see: Gielen, Amy-Jane. "Countering Violent

Extremism: A Realist Review for Assessing What Works, for Whom, in What Circumstances, and How?" Terrorism and Political

Violence (2017): 1-19.

IE those short of legal or security measures such as arrests, freezing of finances, police surveillance, etc.

Usually measured in terms of beneficiaries reached, program retention/completion rates, and other metrics that indicate

whether the program operates as designed and reaches the intended participants. Some programs do include participant

feedback (either qualitative in the form of focus groups, or quantitative, usually in the form of evaluation surveys), but this

feedback usually revolves around similar measures which Gielen (2017) describes as “process evaluation.”

Ibid (p. 6)

Thomas, Paul. "Failed and friendless: the UK's ‘Preventing Violent Extremism’ programme." The British Journal of Politics and

International Relations 12.3 (2010): 442-458.

Schanzer, David H., et al. The challenge and promise of using community policing strategies to prevent violent extremism: A

call for community partnerships with law enforcement to enhance public safety. Durham: Triangle Center on Terrorism and

Homeland Security, 2016.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 11

A second major challenge to studying P/CVE program impact is with respect to program design and

implementation itself: at what level of aggregation should programming be targeted? And which stage of

radicalization or VEO involvement should be addressed by the program? These issues are examined

further in the next section of this report, with the case of Bangladesh in mind. A third major

challenge is in measurement of program effects: even if a P/CVE program is correctly implemented as

designed and reaches its target audience, how would we know whether participants receiving

“treatment” are actually being affected by it in ways consistent with the program’s policy goals?

Outcomes of interest (which include concepts such as radicalization and the risk of participating in

violent extremism/violent extremist groups, or violent extremism antecedent attitudes/behaviors

) are

rare and difficult to detect in the form of observable behaviors. Attitudinal measures (i.e. surveys) must

contend with social desirability bias, which can present challenges when using them for measuring

program effects.

In summary, the paucity of convincing evidence for the efficacy of P/CVE programs is not due to

incompetence or carelessness on the part of researchers and practitioners, nor is it due to poorly-

designed programming. Rather, it is due to these aforementioned methodological challenges and the

“needle-in-the-haystack” nature of the substantive problem itself. Despite these challenges, some notably

well-designed studies have been helpful in clarifying areas where P/CVE programming might be most

promising. Work by Aldrich (2012; 2014) studying USAID radio programs in Mali, Chad and Nigeria, for

example, has provided quasi-experimental evidence that CVE messaging can alter norms of civic

participation and behavior among individuals in ways suggesting a greater willingness to work with the

West in combatting terrorism.

An evaluation by Williams et al. (2016) has provided evidence that community-focused secondary

prevention programs may be effective when targeted towards peer “gatekeepers.”

Given the difficulties in implementing and assessing efficacy of programs that attempt to identify at-risk

individuals and/or intervene in cases further down the radicalization pathway, a more promising

approach for P/CVE programming is likely in the area of prevention. Prevention can be achieved through

counter-communication, resilience-building, and/or the provision of support to key networks and actors

who are likely most efficacious in prevention efforts.

A useful taxonomy of P/CVE prevention

Attitudes and behaviors that are often associated with those who later go on to participate in violent extremist groups

and/or acts of political violence. These attitudes may be context-dependent in their specific manifestation, but often have

common themes which include (but are not limited to) out-group disgust (i.e. racism, homophobia, anti-Semitism, etc.),

misogyny, authoritarian personalities and views, a need for closure, or black-and-white worldviews. Associated behaviors might

include, for example, hate speech, violence against members of the out-group (or the advocacy thereof), intolerance for

differing beliefs/worldviews, or risk-taking behavior that is indicative of fatalism. It is important to note that violent extremism

antecedent attitudes/behaviors are not ubiquitous among those involved in violent extremism, nor are those who exhibit such

attitudes/behaviors certain to radicalize and pursue violence. For a further discussion, see: Gambetta, Diego, and Steffen

Hertog. Engineers of Jihad: The curious connection between violent extremism and education. Princeton University Press, 2017.

This problem is discussed at length by Horgan & Braddock (2010) in evaluating “de-radicalization” programs, which are often

implemented in such a way that incentivizes participants to exhibit reformed attitudes and behaviors, often to avoid punishment

or secure better prison conditions, early release, etc.; see” Horgan, John, and Kurt Braddock. "Rehabilitating the terrorists?

Challenges in assessing the effectiveness of de-radicalization programs." Terrorism and Political Violence 22.2 (2010): 267-291.

Aldrich, Daniel P. "Radio as the voice of god: Peace and tolerance radio programming’s impact on norms." Perspectives on

Terrorism 6.6 (2012): 34-60; Aldrich, Daniel P. "First steps towards hearts and minds? USAID's countering violent extremism

policies in Africa." Terrorism and Political Violence 26.3 (2014): 523-546

Williams, Michael J., John G. Horgan, and William P. Evans. "The critical role of friends in networks for countering violent

extremism: toward a theory of vicarious help-seeking." Behavioral Sciences of Terrorism and Political Aggression 8.1 (2016): 45-65.

Harris-Hogan, Shandon, Kate Barrelle, and Andrew Zammit. "What is countering violent extremism? Exploring CVE policy

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 12

programs is inspired by public health models, and separates such efforts into those deemed primary

prevention (aimed at decreasing “new cases,” or incidence of radicalized individuals), secondary

prevention (aimed at decreasing the prevalence of radicalized beliefs within a network and dissuading

radicalized individuals from going on to potentially engage in violent extremism), and tertiary prevention

(aimed at convincing those already engaged in violent extremism to disengage and de-radicalize).

Notably, this model of radicalization remains agnostic to the potential actions taken by radicalized

persons (which are not necessarily violent). Moreover, not all of those who commit acts of violence on

behalf of VEOs have been radicalized; however, for non-conflict zones (and certainly the case of

Bangladesh), radicalization is generally prerequisite to involvement in (and violent acts on behalf of)

VEOs.

Regarding this taxonomy of prevention-focused P/CVE programs, tertiary efforts pose the greatest

difficulty for targeting and for evaluation, due to the methodological reasons discussed above. While

“soft” programming that targets individuals already involved in violent extremism likely has an important

place in policymakers’ repertoire of P/CVE tools, such efforts in practice are usually targeted to those

who have already had contact with state security efforts (i.e., prisoners), and thus fail to reach those

currently engaged in violent extremist activities for the obvious reason that such individuals are doing so

clandestinely or are not otherwise reachable.

Primary and secondary prevention efforts form the bulk of prevention-focused P/CVE programming

attempted thus far. The sparse evidence available from past evaluation paints a mixed picture regarding

the efficacy of primary prevention P/CVE efforts, i.e., those designed to “inoculate” a population against

extremist ideology through programs that address root causes of radicalization. These primary

prevention programs are usually attempted through direct counter-messaging efforts, or indirectly by

addressing psycho-sociological causes of radicalization, by imbuing critical thinking skills, or by raising

awareness among families and support services. For example, several studies have more or less

concurred that interventions aiming to address inter-group prejudice or extremism can be effective in

changing norms of behavior, though not necessarily in changing the underlying attitudes and beliefs of

individuals.

Such findings suggest that prevention-focused P/CVE programs designed to persuade those

already holding violent extremist antecedent attitudes, or who may sympathize with violent extremist

groups, are likely to be ineffective, particularly if the program is short term and/or of low duration.

Unlike efforts to change attitudes and beliefs, relatively well-documented findings are available to suggest

that primary P/CVE interventions can change behavioral norms in areas substantively related to violent

extremism. This evidence comes from evaluations of programs designed to combat racism, gender

violence, and school bullying (among others).

The hypothesized mechanism behind such programming

and practice in Australia." Behavioral Sciences of Terrorism and Political Aggression 8.1 (2016): 6-24

Ibid.

Examples are varied, and include studies of interventions focusing directly on attitudes towards violent extremism (Aldrich

2012; 2014), as well as those meant to address antecedent behaviors such as out-group prejudice, and violence against women

(see: Scacco, Alexandra, and Shana S. Warren. "Can social contact reduce prejudice and discrimination? Evidence from a field

experiment in Nigeria." American Political Science Review (2018): 1-24; Coker, Ann L., et al. "Evaluation of Green Dot: An active

bystander intervention to reduce sexual violence on college campuses." Violence against Women 17.6 (2011): 777-796.) Doubts

about the efficacy of persuasion attempts via short and medium-term programming are apparent in substantive foci beyond CVE

and antecedent behaviors as well; regarding attitudes on dating violence, for example, Storer et al. (2016)’s review of 15

programs found mixed evidence that bystander intervention programming reduced rape myth acceptance among men despite

being effective at changing behavioral norms.

See, for example, Coker, Ann L., et al. "Evaluation of Green Dot: An active bystander intervention to reduce sexual violence

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 13

is that, by publicizing and reinforcing a norm against some socially undesirable behavior, it both decreases the

likelihood of its expression among those holding attitudes that might lead them to express it, and empowers

others who hold attitudes opposed to the behavior to further reinforce the norm against it. Applying such

findings to primary prevention-focused P/CVE programs suggests that the most promising approaches

might focus on changing the social milieu of acceptable behaviors regarding violent extremism

antecedent behaviors, rather than attempting to directly change individual-level attitudes.

An obvious impediment to primary prevention-focused P/CVE efforts is one of program scale, given that

the potential social reservoir for “at-risk” individuals covers a broad swath of the population, even when

accounting for some regularities among the eventual perpetrators of violent extremist acts.

Moreover,

while such efforts may be feasible in contexts with available resources to devote to wide-spread

implementation of such programming, the very correlates that put a society at high risk of violent

extremism- challenges such as unequal development and income inequality, political polarization, social

cleavages, distrust in the criminal justice system, and strained social support services- also leave it

unequipped to implement such measures at scale. Take the public health model of prevention, for

example: deploying such efforts in a resource-limited context would be like implementing a hand-

washing campaign in a society that lacks clean water infrastructure and soap.

Given the targeting and methodological impediments to tertiary prevention CVE programs, and the

practical impediments to primary prevention, secondary prevention CVE programs are likely to offer the

most cost-effective and potentially fruitful approach to addressing violent extremism in a context such as

Bangladesh. Such efforts attempt to prevent further radicalization of individuals holding violent

extremism antecedent attitudes, while empowering individuals within the networks of those at-risk to

act upon witnessing violent extremism antecedent behaviors and therefore reduce their expression and

diffusion. Specifically, an application of the BIT model for secondary prevention might offer a promising

and practical method for CVE in a context such as Bangladesh. The next section of this report details

the BIT model and its adaptation for secondary P/CVE purposes in Bangladesh.

4. THEORY OF CHANGE

This evaluation considers a capacity-building secondary P/CVE strategy for creating social environments

hostile to violent extremism, focusing on promoting the social norm of “tolerance” through bystander

intervention training (BIT). Furthermore, the evaluation considers a BIT strategy focusing on “peer

gatekeepers,” both as the individuals delivering the training, and those receiving it. As discussed,

programs that attempt to target at-risk youth are usually of limited success because they require finding

the proverbial needle in the haystack. A more effective strategy, then, may be to change the permissive

social milieus in which hateful speech, harassment, and other antecedent behaviors of violent extremism

are left unchallenged, leaving an open path for escalation toward actual violence. The principal

investigators therefore proposed testing programs to change the social environments that may allow

on college campuses." Violence against Women 17.6 (2011): 777-796.)

It is well-known, for example, that most individuals who commit acts of terrorism are young males, with some evidence

suggesting that educated and middle-class males may be especially vulnerable, and even more so if they experience “significance

loss events.” However, even with slightly better specific demographic and psychological predictors- the observation in

Bangladesh, for example, that perpetrators tend to come from the educated middle class- the target population for primary

prevention CVE programming would be too vast to realistically and efficiently reach.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 14

violent extremism to grow, delivered by those whose identities or social positions among the target

audience would potentially facilitate said changes.

We ask:

1. What is the effect of a tolerance-focused bystander intervention training P/CVE program on

effective intervention to promote and protect norms and behaviors against P/CVE antecedent

attitudes and behaviors among those receiving it?

2. Are trained peer leaders more effective at delivering said bystander intervention training program

than traditional authority figures for the context?

We theorize that the tolerance BIT program will affect change through two pathways. First the program

will change norms within the subject pool to reject hate and be more tolerant of various outgroups,

such as Hindus, women, and Westerners. Second, the program will increase subjects’ capacities to take

pro-tolerance actions and make the subjects more effective as pro-tolerance interveners by training

them in effective social intervention techniques.

The first pathway in the theory of social change is through improving norms of tolerance. The use of the

term norm is unfortunately quite varied, referring sometimes to how the members of a group do behave

and other times to how such members should behave. The former type of norm, how members of a

group do behave, is called a descriptive norm and the latter type of norm, describing how members of

the group should behave, is called a prescriptive norm.

In theory individuals comply with norms because

failure to do so risks some form of social censure (for example embarrassment, reprimand and

exclusion). Psychologists frequently argue that individuals are often imperfectly informed about actual

norms due to limited opportunities for observation and cognitive shortcomings.

Thus, psychologists

are often more interested in perceived norms rather than actual norms. We follow that convention in

this report: when we use the term norms we are actually speaking of perceived norms. Most of our

analysis focuses on prescriptive norms, although we do not ignore descriptive norms entirely.

Norms are to be distinguished from what psychologists call attitudes but what economists might call

preferences. Attitudes/preferences are an individual taste for an activity. A person who engages in

tolerant behavior because they like pleasant social interactions or were born with an elevated sense of

empathy are indulging a preference for tolerance. Programs typically attempt to bring about social

change by changing perceived norms rather than attitudes/preferences because the latter are thought to

be harder to change than the former.

Compliance with perceived norms is undoubtedly aided by a

preference for the normative behavior but it is not necessary. Compliance with perceived norms

operates through a desire for social acceptance. For example, people may recycle even when they find it

Prentice, Deborah A. 2007. Norms, Prescriptive and Descriptive. Encyclopedia of Social Psychology, Roy F. Baumeister, and

Kathleen D. Vohs, Sage Publications, 1st edition.

Tankard Margaret E. and Elizabeth Levy Paluck. 2016. Norm Perception as a Vehicle for Social Change. Social Issues and Policy

Review. Vol. 10:1 181—211; Perkins, H. W. (2002). Social norms and the prevention of alcohol misuse in collegiate contexts

Journal of Studies on Alcohol, Supplement, 14, 164–172 ; Perkins, H.W.,Meilman, P.W., Leichliter, J. S., Cashin, J. R., & Presley, C.

A. 1992). Misperceptions of the norms for the frequency of alcohol and other drug use on college campuses. Journal of American

College Health, 47(6), 253–25.

Tankard Margaret E. and Elizabeth Levy Paluck. 2016. Norm Perception as a Vehicle for Social Change. Social Issues and Policy

Review. Vol. 10:1 181—211

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 15

inconvenient for fear of social sanction if they do not. Thus, changing perceived social norms can affect

broad social change even when underlying attitudes or preferences remain fixed.

Tankard and Paluck (2016) list three sources of information that can change perceived norms: the

behavior of salient individuals, descriptive information about the community and institutional signals.

They give the example of a hypothetical program to increase recycling in a community: “…a norm

change intervention may, for example, expose people to a popular peer who recycles, provide people

with information that most of their peers recycle, or advertise new recycling guidelines from an

important and trusted community institution.” (p. 183).

Recent research has indicated that changing individuals’ perceived norms can successfully produce

changes in individuals’ behavior and broader social change through these three channels. For example,

Paluck and Shepherd (2012)

and Paluck, Shepherd, and Aronow (2016)

report on an intervention

that reduced high-school bullying by changing perceived norms through the first channel mentioned

above: popular peers. Perkins and Craig (2006) employed the second channel, information about average

peer behavior, to reduce alcohol abuse in college.

Tankard and Paluck (2017) show how a Supreme

Court decision changed the norm on same-sex marriage via the third channel, i.e., a trusted community

institution, even when underlying attitudes on same sex marriage had not necessarily changed.

The BIT program studied in this report aimed to change perceived norms through the first channel, i.e.,

through statements/behavior of salient individuals (the trained facilitators). We tested the ability of two

sorts of salient individuals to convey the message of norms of tolerance: peer leaders and college

instructors.

We propose that effective intervention (i.e., the promotion and protection of desirable norms- in this

case norms of tolerance and norms against violence) requires individuals to have internalized the

prescriptive norms of tolerance, while also having both the competence and the confidence to publicly

defend those norms in the face of hateful speech or actions. An individual’s confidence and competence

to promote and defend prescriptive norms, in turn, is predicated on (1) the individual’s internalization of

those norms, and (2) the internalization of capacities (i.e., skills) necessary to safely promote and defend

the norms.

We expect both a tolerance-focused bystander intervention curriculum, coupled with a

Ibid

Paluck, E. L., & Shepherd, H. (2012). The salience of social referents: A field experiment on collective norms and harassment

behavior in a school social network. Journal of Personality and Social Psychology, 103, pp. 899–915.

Paluck, E. L., Shepherd, H., & Aronow, P. 2016. Changing climates of conflict: A social network driven experiment in 56

schools. Proceedings of the National Academy of Sciences

Perkins, H. W., & Craig, D. W. (2006). A successful social norms campaign to reduce alcohol misuse among college student-

athletes. Journal of Studies on Alcohol, 67(6), 880–889.

Tankard M.E., Paluck E.L. (2017). The Effect of a Supreme Court Decision Regarding Gay Marriage on Social Norms and

Personal Attitudes. Psychological Science. 28(9):1334-1344.

Note that the internalization of desirable (i.e. tolerant) norms is considered prerequisite to skill-building too. This reflects the

fairly straightforward supposition that a person will not intervene to promote/protect a social norm if they themselves have not

internalized it. Indeed, we expect that the majority of participants in a program of this nature will either (1) already have

tolerant norms, or (2) will be open to more tolerant views. We do not expect individuals with intolerant viewpoints to remain

in a program of this nature by choice.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 16

trained facilitator implementing programmatic elements of the curriculum, to promote the

internalization of norms and skills necessary for effective intervention.

Our second proposed pathway is to build individual citizen’s capacity to safely confront and discredit

hateful extremist views so that those who hold extremist views know that society will not tolerate

them. We expect that this outcome (which we dub “willingness to intervene”) requires both the

internalization of social norms (discussed above; in this case of tolerance) as prerequisite to an

individual’s choice to intervene on behalf of those norms, and the internalization of capacity-building

skills required to intervene. We expect that the internalization of norms and skills needed for

willingness to intervene is driven both by participant exposure to a program designed to impart and

teach those respective norms/skills, as well as the participant’s “buy-in,” or the extent to which they

identify with the program goals, with those of their co-participants, and with the facilitator leading the

programming. Finally, we expect that the use of program-taught skills for willingness to intervene

requires a participant’s confidence and competence to actually act, should they find themselves in

circumstances where willingness to intervene is possible. We visualize the aforementioned set of

pathways from programming to willingness to intervene in Figure 1.

Figure 1: Pathways from BIT Programming to Willingness to Intervene

Programs based on the first proposed pathway—cultivating social norms of tolerance—have many

models. One approach is the Teaching Tolerance curriculum of the Southern Poverty Law Center

(http://www.tolerance.org)

. Teaching Tolerance is a literacy-based program that can be folded into

elementary and high schools’ existing curricula. Their curriculum is targeted toward an American

audience to address racism, religious intolerance, and anti-immigrant sentiment. However, we believe

we have adapted it appropriately to the Bangladeshi social context in order to spread norms of

tolerance, non-violence, and acceptance of secularism and other differences.

A second approach to changing norms is the Roots curriculum developed by Princeton Psychologist

Elizabeth Levy Paluck and her colleagues.

The curriculum was designed to reduce bullying in American

See http://www.betsylevypaluck.com/roots-curriculum/.

Internalize

Norms

Tolerance BIT

Curriculum

Facilitator

Identity

Internalize

Skills

Confidence/

Competence

to Act

Effective

Intervention

on Behalf of

Victim

Identification

with Facilitator

and Program

Goals

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 17

high schools, but the social theory behind it is widely applicable to many undesirable social behaviors,

including extremism. For this program, the social networks of schools were mapped. Important “nodes''

in this social network—people with many friends, or people who are widely admired by their peers—

are enlisted and taught how to spread the desired norm. The theory behind the program is that these

important nodes in the social networks are norms creators and spreaders, and their important social

status can be leveraged to spread the desired norms and reduce antisocial behavior. A challenge to using

this model for P/CVE purposes is that it requires the mapping of an established social network. Our

target audience—first and second-year university students—included a large number of participants who

had not yet established their own social networks in a new setting, therefore rendering a process of

identifying central nodes, similar to that of the Roots program, difficult.

Moreover, the university

context of Bangladesh presents additional complications, given that student leaders are sometimes

affiliated with student political groups known for violence and hazing rituals. Because we were interested

in comparing the efficacy of peer leaders to that of traditional authority figures for the context, we

deferred to university officials working with our implementation partner (Rupantar) to identify upper-

level university students with prior leadership experience (which is not tied to partisan organizations)

for this purpose.

Programming based on the second pathway—improving capacity and skills to safely intervene and

counter hateful and violent extremist speech—can be modeled on a variety of “bystander training”

programs developed for American campuses and high schools. In a randomized study in 26 Kentucky

high schools over five years, one such program, The Green Dot (https://www.livethegreendot.com)

, was

shown to reduce multiple forms of interpersonal violence.

The program includes schoolwide training

on bystander intervention by Green Dot educators and, like the Roots programs described above, it

leverages student popular opinion leaders to change norms. StepUP (http://stepupprogram.org/)

developed at the University of Arizona, is probably the most widespread and commonly used program

on college campuses in the United States. (The Active Zone bystander intervention program at NYU,

where we work, is based on it). We are unaware of any randomized studies of the program, but it has

been labeled “best practice” by the NCAA (National Collegiate Athletic Association), and has won the

NASPA (Student Affairs Administrators in Higher Education) Gold Award. The program teaches a

simple five-step decision making process for bystander intervention that should be easily translatable

into the Bangladeshi context.

FACTORS INHIBITING THE IMPACT OF BYSTANDER INTERVENTION TRAINING

Conditions in the study sites in Bangladesh are such that the BIT program will not emphasize identifying

and connecting at-risk individuals with police or other P/CVE service providers. Even in the US, studies

have found that while friends of at-risk individuals are usually best positioned for noticing early signs of

violent extremism,

they are typically reluctant to connect the at-risk person to police or P/CVE service

providers.

A series of studies on this issue found that the reasons cited for this reluctance mainly

We found in baseline surveys, for example, that many students had difficulty naming five friends within their university who

would potentially be interested in future iterations of the programming.

Coker AL et al. 2017. RCT Testing Bystander Effectiveness Training to Reduce Violence. American Journal of Preventive

Medicine. 52:566-78 (gated link to the study).

During December 2019 site visits with university officials, we found that program trainers and small groups of trainees

believed that the “5 D’s” of bystander intervention training were easily understood and remembered by program trainees.

Williams et al., (2016a)

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 18

centered on a fear of damaging their relationship with the person, fear of getting the person into

trouble, and fear of getting into trouble themselves, should they attempt to connect the person with a

CVE provider.

Therefore, any programming that focuses on training peer gatekeepers should account

for this reluctance in its design.

We would assume that such reluctance would be relevant in our

Bangladesh sites as well.

The bulk of research on secondary P/CVE programs has taken place in North America and Western

Europe, where P/CVE service providers have been able to establish themselves as trusted intermediaries

between communities and law enforcement. Such programs are often implemented as a bundle of

interventions, which may include community education, volunteering and multicultural programming, in

addition to cultivating resilience against extremism and building ties between communities and law

enforcement.

The success of a P/CVE service provider model is predicated on an assumption that

neither the referring gatekeeper, nor the at-risk referent, will face punishment or stigma (either by their

own communities or the criminal justice system/security apparatus), should an intervention take place.

Unfortunately, this assumption is unlikely to hold in a context such as Bangladesh for a variety of

reasons, including a closing civic space, a zero-tolerance approach by security forces towards those

expressing sympathy with extremist groups, and widespread stigma around issues of mental health. Even

if trusted secondary prevention CVE providers were to be available in this context, well-positioned

gatekeepers such as friends would likely be even more reluctant to connect at-risk individuals with the

providers for these reasons. When accounting for the lack of such providers, reluctant gatekeepers may

also feel powerless to render help to an at-risk friend, given that doing so may result in extreme

consequences for themselves and the friend.

The aforementioned trade-offs between a gatekeeper’s social proximity to an at-risk person and their

reluctance to intervene due to fear of repercussions suggests that adopting a capacity-building program

to a context such as that of Bangladesh will require re-thinking what constitutes realistic and appropriate

forms of intervention/assistance, and how to train individuals to safely implement a decision to assist

that accounts for risks to themselves and the person they’re attempting to help. In contexts of extreme

reluctance among potential gatekeepers, a best alternative approach may be to empower bystanders to

engage directly and privately with those they notice are at-risk of violent extremism, while providing

them with an awareness of resources that can help them offer instrumental, informational, affiliative, or

emotional support—hence the potential value of a bystander intervention training program, which is

designed to train participants how they can respond as individuals to “social emergencies” without the

immediate availability of authorities/service providers.

BIT can also equip individuals to safely signal to

their peers that they themselves can serve as a confidential resource for those concerned about others,

either by helping them to accurately identify warning signs, or to steer them towards an intervention

plan.

Ibid; Williams et al. (2016a) note that variation in reluctance likely exists among individuals by personality type and

attachment style.

The BIT program we designed for Bangladeshi universities does not rely on P/CVE “service providers.” In addition to the

peer reluctance issues, our scoping missions found that very few potential P/CVE service providers exist for the context.

A prominent example being the World Organization for Resource Development and Education (WORDE) “portfolio

approach” to countering violent extremism, which consisted of several interlocking programs; see evaluation by Williams et al.

(2016b).

Ibid.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 19

Finally, the challenges of social and political contexts like those of Bangladesh require a focus on helping

individuals at earlier stages of the radicalization process than those in Western contexts, due to the

risks of stigma and/or harm to both a bystander who intervenes, and to the individual they are trying to

help. It is well-established, for example that those who sympathize with or go on to support and join

violent extremist groups frequently (though not always) develop and exhibit attitudes and behaviors that

dehumanize/express disgust towards targeted social groups, and ascribe blame and sanctioning to

members of that outgroup for perceived transgressions. Other common antecedent attitudes/behaviors

include sentiments of in-group victimhood, collective narcissism, and/or in-group nationalism. Violent

extremist groups both promote the development of such attitudes in their messaging, and recruit from

those exhibiting them. Thus, in a context where P/CVE interventions focusing directly on those who

have radicalized and are at high risk of violent extremism are unlikely to be fruitful. A better alternative

may be to focus on combating norms of identity-based intolerance and promoting norms of out-group

tolerance.

ADAPTING THE BYSTANDER INTERVENTION MODEL TO REDUCE VIOLENT EXTREMISM

ANTECEDENT BEHAVIORS

The classic model of bystander intervention (focusing on medical emergencies), described by Latane and

Darley (1968), includes five steps from the occurrence of an emergency to a person’s intervention in

order to render assistance. These steps include (1) noticing the event, (2) interpreting it as an

emergency, (3) assuming responsibility to act, (4) knowing an appropriate form of assistance, and (5)

implementing a decision to assist. Each of these steps pose challenges which may lead to a failure to act.

The well-known “bystander effect,” for example, is caused by a failure of individuals to assume personal

responsibility to act, which increases in likelihood with the presence of more bystanders (i.e., the

diffusion of responsibility). The goal of BIT programs is to train individuals to recognize these sources of

intervention failure, take responsibility for, and then safely and effectively respond to problematic

situations in lieu of official help channels. In order to achieve this goal, the training usually focuses on

addressing impediments to acting by empowering individuals with a toolkit of potential actions they

could take under various circumstances.

A key advantage of adopting a BIT model for P/CVE purposes is that it focuses on intervening in ways

that are safe. BIT explicitly addresses and prioritizes safety for trainees and others, by teaching trainees

how to evaluate the urgency and risks posed by different situations in which they might find themselves.

Although BIT is designed to teach trainees to overcome the “bystander effect,” it does not encourage

trainees to engage in “heroic” behaviors which would put themselves (or potentially those they are

trying to help) at risk of injury that can result from escalating a situation.

The training usually

discourages the use of direct confrontation unless the bystander has safety in numbers, and instead

A canonical example of such a program for medical emergencies is training provided by the American Red Cross, which

teaches bystanders how to recognize a medical emergency, assess the safety of the situation/area, direct others in the area to

provide support by alerting emergency services, clearing a location, etc., and then rendering first aid until help arrives. For

versions adopted to preventing sexual assault/harassment, this protocol might take the form of recognizing aggressive behavior

at a party, assessing whether it has the potential to escalate, recruiting assistance from others nearby if available, and then

intervening usually by distracting the perpetrator and then checking in with the victim once the incident ends.

The issue of safety and potential escalation was also a key area of focus during September, 2019 facilitator training sessions.

For the context of Bangladesh, facilitators noted that caveats applied to the use of some of the “5 D’s” strategies. For example,

the “documenting” strategy (in which a bystander documents evidence of a perpetrator’s actions for potential later use in legal

proceedings) might be inappropriate to use under circumstances when perpetrators have connections or influence over local

security authorities.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 20

promotes de-escalatory strategies such as the creation of a distraction in order to interrupt a

perpetrator’s behavior before it becomes dangerous.

An additional advantage of adopting a BIT model for secondary prevention P/CVE in a context such as

Bangladesh is that it does not presume or require the availability of P/CVE service providers, which

generally have been the crux of community-focused secondary P/CVE programming implemented in

Western democracies. Given shrinking civic space, crackdowns by the state security apparatus, and

enduring stigma towards the discussion and treatment of mental health issues, a secondary P/CVE model

premised on gatekeeper referrals to P/CVE service providers in the Bangladeshi context would likely be

ineffective at best, and potentially dangerous to both referents and referees at worst. As discussed, prior

findings (in the US) suggest that peer gatekeepers would hesitate to refer an at-risk friend to support

services due to a fear of harming their relationship with the friend;

such concerns would likely be

exacerbated within the Bangladeshi context. Thus, a BIT model could provide individuals with a toolkit

of approaches they can use upon witnessing incidents of violent extremism antecedent behavior that do

not necessarily require involving third parties, or escalation beyond the immediate incident at hand. The

goal of such a program would be to empower individuals to recognize and safely respond to violent

extremism antecedent behaviors, focusing in this case on the antecedent behaviors of intolerance

towards out-groups.

We detail in the following sections how we designed an intervention that adopted

a tolerance-focused BIT program to the Bangladeshi context and implemented it as an RCT.

5. IMPACT EVALUATION METHODOLOGY

The main purpose of this IE is to estimate and compare the effects of the aforementioned BIT P/CVE

strategy for (1) reducing the support for and expression of violent extremist antecedent behaviors, and

(2) promoting the safe exhibition of pro-social behavioral norms in order to create social climates

hostile to violent extremism in the form of bystander intervention. We pursued this goal through three

steps, detailed in this section of the report. First, we partnered with a Bangladeshi NGO (Rupantar) to

adopt a tolerance-focused BIT program for Bangladeshi university students and train university-

nominated peer and authoritative influencers to deliver it. As a feature of the experimental design, we

also assisted Rupantar in developing a placebo curriculum of a similar duration/pacing as the BIT

treatment. Furthermore, during April 2019 design meetings with Rupantar, we trained members of the

implementation team to implement RCTs.

Second, after working with Rupantar to develop a Bangladeshi BIT curriculum and train facilitators to

correctly implement it, we conducted a randomized, placebo-controlled trial featuring three treatment

statuses over two treatment waves into which individual participants were recruited and randomized, as

follows:

Williams et al. (2016a).

“Out-groups,” in this case, are defined as those which have experienced recent violence and targeted hate speech in

Bangladesh. These have included (but are not limited to): Hindus, Buddhists, Christians, Shia Muslims, Atheists, LGBTQ

persons, Westerners/foreigners, and those holding secular and/or socially liberal political attitudes.

These training sessions were extremely helpful to ensuring that the experimental protocol was followed. Namely, Rupantar

team members followed the randomization protocol closely and there were fewer than 5 participants (out of more than 1400)

who were enrolled within a different treatment group than that to which they were randomly assigned.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 21

• Treatment 1: Delivery of a secondary P/CVE program in the form of a tolerance-focused BIT

youth leadership program via top-down authoritative influencers,

• Treatment 2: Delivery of a secondary P/CVE program in the form of a tolerance-focused BIT

youth leadership program via horizontal peer influencers,

• Placebo: Delivery of a youth leadership program focused on challenges of development within

Bangladesh, led by authoritative influencers as in Treatment 1 above.

• No-Program Control (Wave 2 only): Given that applicants to the program exceeded available

slots (and that many interested students could not participate due to schedule conflicts), we added a

no-program control to the study during wave 2 incentivized with a small gift card to a popular online

merchant. Unfortunately, recruitment for the no program control was disrupted due to country-

wide closures of the university system in late March, 2020 because of the Covid-19 pandemic. We

discuss implications in subsequent sections.

Third, in order to measure intervention efficacy, participants across all treatment statuses completed

baseline and endline surveys that included measures related to norms of tolerance, radicalization

antecedent attitudes, and views on violent extremism. In addition, the surveys included several vignettes

designed to elicit whether and to what extent participants would report their likelihood of reacting to a

situation presented to them using one or more bystander intervention tactics. The surveys also included

perceptions and evaluations of the participant’s facilitator, which are used in standard course

evaluations. Moreover, information was collected on participants’ demographic and socio-economic

characteristics, as well as a battery of psychosocial and community characteristics at baseline. The

following subsections detail the intervention design, site selection, participant recruitment, and

randomization procedure. We discuss our measurement strategy and data collection in subsequent

sections.

INTERVENTION DESIGN: ADAPTING BIT TO BANGLADESHI UNIVERSITIES

What should bystander intervention training to promote tolerance and deter hateful, intolerant

behaviors and speech look like in the Bangladeshi context? There were no analogous initiatives within

Bangladesh at the time of program development; the consideration of P/CVE issues and violent

extremist antecedent behaviors have largely been considered criminal justice matters until recently.

order to design the programming, we drew on content from existing publicly available BIT programs

that have been implemented within U.S. universities and other youth settings, and worked closely with

Rupantar to adapt the curriculum for university students in Bangladesh. Rupantar is a well-established

NGO working within Bangladesh, and was chosen as implementation partner for the programming due

to their extensive prior experience working within the university system, as well as their ability to scale

for reaching sufficient enrollments for conducting the program as an RCT.

This became a clear theme to us from November, 2017 and June, 2018 interviews with multiple representatives from NGOs

and universities engaged in programming designed to build confidence and promote mental health among young people.

Although a variety of NGOs are engaged in programming meant to, for example, built leadership skills among Bangladeshi

youth, most are focused on building career-oriented skillsets. While mental health programming is becoming more common in

Bangladesh, there are still social taboos associated with discussing mental health issues and how they might be associated with

vulnerability to radicalization.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 22

The final version of the tolerance BIT program

consisted of 14, two-hour sessions, for a total of 28

hours of programming. Our evaluation of evidence for the efficacy of BIT programs targeting different

anti-social behaviors in universities suggested that the frequency and duration of such programs matter

for retention and application of the training. The vast majority of programs targeting, for example, sexual

harassment and dating violence, only seem to show long-term impact among participants when the

programming is implemented across multiple sessions.

In most cases, participants met twice a week for the sessions, although exact schedules varied by

university and facilitator. The programming was designed to accommodate early drop-outs and

replacements, as well as the measurement strategy we devised for evaluating efficacy. The bulk of the

substantive programming was delivered during sessions 3-11. Early university students (in their first or

second-year) received the programming in small/medium-group format, with classes capped at 20 and

usually ranging from 10-15 after accounting for attrition.

The structure of the curriculum itself focused on developing (1) domain knowledge development

through lecture and class discussion about the issues of tolerance/intolerance within Bangladesh,

followed by (2) bystander intervention training. Because the program was designed as an RCT with a

placebo, the curriculum was marketed as a “youth leadership training program” and also integrated

sessions meant to develop generic leadership skills (such as public speaking and group presentations). A

short summary of the topics of focus for each of the 14 sessions is as follows:

• Sessions 1-2: Leadership training (i.e., public speaking skills, group presentations, debate)

• Sessions 3-6: Domain knowledge training (intolerance within Bangladesh)

• Sessions 7-11: Bystander intervention training

• Sessions 12-14: Application of BIT module via activities (role-playing, implementation

intentions, skits).

Finally, a separate placebo curriculum was developed with the help of Rupantar. The placebo had

participants meeting for the same frequency/duration in small groups with facilitators. However, the

placebo curriculum focused on the challenges associated with rapid development in Bangladesh, including

transportation issues (i.e., urban traffic), environmental issues (i.e., pollution), and public health issues

(i.e., smoking and fast food). We were careful during the design of the placebo to avoid creating content

that would promote focused discussion on intolerance and/or issues related to P/CVE. Moreover,

Rupantar was well aware of the importance of this as a key feature in the experimental design, following

our training sessions with them. Joint curriculum development and training sessions with Rupantar

occurred during and after our first meetings with them in April, 2019.

The detailed curriculum is available in the appendices of this report, and separately in the document entitled.

We were aware of a multitude of other P/CVE programs implemented under the Obirodh initiative as shorter duration

programs- usually in the form of single, day-long workshops. We learned early during scoping missions that various self-

improvement workshops are extremely common, and that they are often competing for limited student attention/time.

Most university classes in Bangladesh are extremely large.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 23

SITE SELECTION, PARTICIPANT RECRUITMENT, AND RANDOMIZATION PROCEDURE

Parallel to curriculum development and training sessions with Rupantar, we solicited interest from

Bangladeshi universities that were (1) interested in hosting the programming within the targeted cities of

Dhaka and Rajshahi,

(2) large enough for sufficient subject enrollment to take place for the needs of

the RCT, and (3) somewhat representative of the diversity of Bangladeshi higher education institutions

and students enrolled within them. A total of 9 schools (5 in Dhaka, 4 in Rajshahi) were formally

approached following earlier scoping missions, or at the suggestion of Rupantar. Memorandums of

understanding (MOUs) were eventually signed with 6 schools (Table 1).

Table 1: Participating Public and Private Universities in Dhaka and Rajshahi

Location Universities

Dhaka Daffodil International University (Private)

Jahangirnagar University (Public)

University of Dhaka (Public)

Rajshahi Rajshahi College (Public)

University of Rajshahi (Public)

Varendra University (Private)

Besides offering large resident student populations from which to recruit participants, the public

universities above host students from across the entire country. While we found that the bulk of

participants were from the Dhaka or Rajshahi divisions, there was meaningful representation (at least

100 participants) from those whose home villages fell within 7 of the 8 administrative divisions of

Bangladesh. Moreover, we were keen to include some of the private universities within the study in

order to attract interest from students from differing socioeconomic backgrounds.

The six schools

above signed MOUs with Rupantar in order to begin implementing the programming beginning

September, 2019.

Trainers (called “facilitators”) were recruited during August 2019 by their respective universities,

subject to constraints on age and career progression that were requested and later verified as a feature

of the research design comparing peer leaders to authoritative figures.

The facilitators were provided

Following consultations with USAID, Rajshahi replaced Chittagong as the second targeted city due to Rupantar’s prior access

to universities within it, and the fact that Rajshahi has experienced problems with violent extremist organizations.

Generally, private universities reflect students from higher socioeconomic backgrounds with local origins. The public

universities, which are heavily subsidized by the government, reflect a much broader (albeit high-achieving) cross-section of the

population due to the national exam system, which provides a merit-based admissions criterion. This was borne out in our data

collection, which revealed participants from a majority of the 64 administrative districts (and all 8 divisions) of Bangladesh.

Missing from the study is representation from those within the post-secondary parochial (madrassa) system, though the

participants study exhibited wide variation in, for example, religiosity and religious conservatism, among other factors we might

expect to distinguish a madrassa from universities.

With input from USAID, we defined “peer leaders” as upper-level university students or recent graduates from the university

under the age of 25 who hold/held positions of leadership within the school. As we discussed, we avoided using a peer-

nomination process in order to choose these individuals due to the influence of student political groups. “Authoritative leaders”

were defined as university professors/lecturers currently employed by the university. Aside from asking participants how they

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 24

with both pedagogical training for moderating small-group student-led discussions,

as well as

substantive training for the programming during a 4-day training session. The training sessions and a

short pilot study of the curriculum content were organized and led jointly by Rupantar and by a member

of the IE team during early September 2019 in Dhaka. Facilitator input was repeatedly solicited during

the training and pilot sessions in order to ensure (1) that the curriculum elements (and BIT especially)

were being properly translated for the target audience, and (2) that the facilitators themselves had

agency over shaping a program that they would later be expected to lead.

Participants were recruited by Rupantar team members using flyers with QR codes leading to a short

online interest form. Assistance was also provided by a small group of students (one from each

university) hired to work as technical assistants for the programming. The programming was officially

marketed as a youth social leadership training seminar for first and second year students. A generic

program summary was made available detailing some of the soft skills training which was embedded into

both treatment and placebo curricula. As an incentive to participate, those who attended at least ten of

the fourteen sessions the program were offered a certificate of completion

, provided by

representatives at NORC at the University of Chicago and New York University. Rupantar also

frequently provided snacks and/or lunch in some cases, while the universities furnished meeting spaces

and other necessary logistics. A voluntary program of this nature was expected to draw those interested

in “leadership training” who had the schedule availability to do so. It is thus expected that the

participants generally reflect those who are prosocial and outgoing among the student body; although,

mitigating this to some extent is the fact that leadership qualities are highly valued soft skills within the

context, which likely explains the widespread interest in the program.

One could imagine a version of

the programming (beyond the capacity of the study) that was mandatory, perhaps implemented as a

required “first year seminar” in which all students were expected to attend. Although we cannot directly

compare the profiles and personalities of participants to those who did not participate, we provide

some comparisons between baseline characteristics of those who finished the program, and those who

did not finish the program (detailed in the baseline characteristics section, below).

We received interest from over 3,400 applicants (about 2,600 of whom were deemed eligible first and

second-year students within the age range) across the six schools over the two waves of programming.

Applicants were then asked to complete a secondary enrollment form, including requisite consent for

research forms, scheduling questions for logistics, and a link to part 1 of the baseline survey,

which

perceived their facilitators (which indicated meaningful differences in perceptions towards the two categories), we also

collected objective demographic/background data on each facilitator (age, gender, highest degree, prior teaching experience,

and current employment). We were able to verify, for example, that the median age of “peer leaders” was 24, versus 36.5 for

the professors.

A separate “teaching guide” was created for the purposes of training in order to assist facilitators in understanding the

differences between lecturing, group discussion, and group activities.

Training sessions with the authoritative figures (i.e., university professors) featured much more back-and-forth discussion

about the appropriateness of certain topics and activities. For example, some time was spent discussing whether to integrate

LGBT issues into the curriculum- the facilitators warned that because LGBT issues remain relatively taboo within Bangladeshi

society, formally integrating LGBT topics into the curriculum would pose an attrition risk. Instead, it was decided that

facilitators would moderate and allow open discussion of these topics as students raised them during sessions, without formally

introducing them in presentation materials or activities.

Facilitators were given some latitude to make case by case basis decisions about attendance requirements. Facilitators were

also tasks with keeping attendance records, which were later provided to the PIs both for use in the IE and for awarding

completion certificates.

Field notes from PI scoping missions and input from both Rupantar and participating universities.

Administered using Qualtrics, an online survey platform.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 25

contained most of the questions regarding demographics, psychosocial characteristics, and other

background questions. Completion of the secondary enrollment form enabled placement into the

randomized lottery for admission into the programming.

In order to facilitate replacing no-shows and early drop-outs (which we expected to be common for a

new program that is competing for the attention of busy university students), applicants were

randomized into one of three ordered admission lists. Rupantar was instructed to offer admission to

participants into the programming in strict order of appearance on this list. Admitted participants who

were ultimately unable to attend the sessions due to scheduling or exam conflicts, who failed to respond

to attempts at contact post-application, or who dropped out after up to two sessions, were quickly

replaced with participants further down the list.

Rupantar was given leeway to replace drop-outs

within the sessions of each facilitator up until the third session, at which point the treatment and

placebo curricula diverged. This strategy was designed to ensure that each facilitator’s sessions were

kept as full as possible, without violating the randomization procedure, and while fulfilling our

commitment to using a fair lottery for participation.

Finally, during wave 2, we opted to add a no-program control (NPC) to the study due to a large number

of interested participants who did not participate in the programming because of scheduling conflicts,

exams, or other logistical reasons. Participation in the NPC was incentivized by offering a small gift card

(500 BDT) to a popular Bangladeshi online merchant Daraz.com for completing baseline/endline surveys.

Recruitment for the NPC was disrupted by the country-wide shutdown of universities due to the Covid-

19 pandemic, resulting in only 204 participants completing endline activities.

SAMPLE SIZE, DATA COLLECTION STRATEGY, AND TIMELINE

We calculated minimum sample sizes of about 310 subjects per treatment condition (assuming attrition)

for minimum detectable effects (MDE) of 0.2 standard deviations in outcome measures. We ultimately

proposed sample sizes of 400 per treatment condition in order to account for potential attrition during

programming and any missing/incomplete data. Through the application process discussed in the prior

section, 2,383 eligible applicants (across all universities and both waves of the study) were randomized

into admission lists. Eventually, 1,662 were assigned to facilitators, meaning that they did not drop out

due to scheduling or exam conflicts, or failed to respond to Rupantar’s attempts at coordinating

schedules. A total of 1,468 attended at least 1 session of the programming, and 1,120 attended at least

ten sessions of the programming. Importantly, 437 and 439 of those assigned to peer and authoritative

treatment statuses completed at least seven sessions of the programming (meaning they were exposed

to the BIT module).

We collected all data electronically

using Qualtrics and Inquisit in order to enable participants to

respond to questions in private. All surveys and other measurement materials were presented in

Attrition rates varied across universities and waves, but averaged around 20%. Most participants who dropped out did so

before session 1 of the programming, and cited schedule or exam conflicts as the reason. For a new program competing for

time/attention of university students, this was roughly what we’d expect.

Our meetings with Rupantar suggested that device/internet access would not be a problem, given that the vast majority of

students at the universities participating in the programming had access to some form of smartphone and internet. We verified

this to be the case in baseline surveys; about 96% of participants reported having access to either a smartphone and/or laptop.

The remaining participants likely completed surveys in university computer labs. Electronic/remote data collection proved

fortuitous given that Covid-19 related university shutdowns and revised IRB protocols would have rendered a paper and/or live

interviewer approach impossible for wave 2 participants.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 26

bilingual (English and Bangla) format.

Participants had the ability to save and return to partially

completed surveys during the survey periods, due to survey length. We were able to collect complete

baseline/endline data from 418 and 422 of those assigned to authoritative/peer tolerance BIT

treatments, respectively, which met our calculated sample size requirements for comparison. Moreover,

we collected data from 406 placebo participants and 204 no-program control participants.

The context and subject matter of questions being asked required the data collection and measurement

strategies to consider and mitigate demand effects and social desirability bias. The latter issue (social

desirability bias) was taken especially seriously, given that participant safety issues also had to be

considered. For example, the political and social context within Bangladesh rendered direct questions

about specific violent extremist organizations or political groups undesirable for inclusion in surveys,

both due to safety concerns for participants and because answers to such questions would likely suffer

from social desirability bias. Although the use of a placebo should account for demand effects

while

mitigating social desirability bias to some extent,

it is possible that participants enrolled in the

tolerance BIT program perceived “tolerant” answers to questions as socially desirable. We discuss how

the measurement strategy accounted for some of these issues in a subsequent subsection on outcomes

(below). Features of data collection protocols to mitigate these issues are described as follows.

In order to mitigate social desirability bias as much as possible within the data collection protocols, all

data collection activities that included sensitive questions and those used as outcomes

were

administered separately from programming. Participants were asked to complete these activities in

private, within a generous time frame. Most participants used their personal devices or laptops to

complete them, although some likely completed them on lab computers or a friend’s device.

The PI’s

made it very clear throughout programming and measurement activities (both to facilitators and to the

participants themselves) that responses they provide in all surveys would be stored in US-based servers,

only accessible by the NYU-based principle investigators. We clarified this in plain (English and Bangla)

language by asking facilitators to tell their students that “no person in Bangladesh will ever see the

answers you provide.”

(This was also printed within all data collection emails/SMS messages used).

During training sessions with the facilitators, the PI’s included a segment on data privacy and

confidentiality, where it was also emphasized that the purpose of participant survey responses was not

to evaluate their individual job performance. Finally, NORC at the University of Chicago hired and

trained several “technical coordinators” at each of the host universities, who were available to assist

participants with technology/access issues regarding data collection activities.

Thus all data collection

activities were directly and exclusively overseen by the PIs, with any ground-level technical assistance

Rupantar translated all measurement materials, while a Bangla-speaking research scientist at NORC at the University of

Chicago assisted with verifying translation accuracies. English and Bangla were juxtaposed within the same survey/measurement

items in order to enable participants (the vast majority of whom had some level of bilingual proficiency) to read in their

dominant language, while gleaning additional clarification/nuance from the second language if needed.

The ability of participants to infer the purpose of measurement activities and thus provide “correct answers” should not

differ between treatment and placebo.

All participants were aware, for example, that they were participating in a civic-focused USAID-funded leadership program.

Part 1 of the baseline, which included demographic and psychosocial baseline characteristics, was completed at the end of

session 1 of programming; this survey was framed as a “background and personality survey.”

Via anecdotal reports from NORC-hired “technical coordinators.”

Students were also informed that their personally identifiable information would be removed from any final data sets,

whereas any publicly accessible data will contain only de-identified data.

Technical coordinators were provided with laptops which could be lent to participants for completing survey activities. The

coordinators were instructed to ensure that the participants could do so privately.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 27

provided by those without a contractual relationship with Rupantar or the trained facilitators. Finally,

participants were also reminded during measurement activities (both in the text and at the beginning of

programming) that their survey/activity responses were not a factor in determining eligibility for the

certificate of completion awarded for participation.

Table 2 summarizes the timing of data collection for each treatment status and wave of the study.

Table 2: Data Collection Timeline

Treatment Placebo

Peer

Tolerance

Prof/

Tolerance Placebo

Peer/

Tolerance

Prof/

Tolerance

Pure

Control

Wave 1 1 1 2 2 2 2

Intake data October 2019

(N=240)

October 2019

(N=243)

October 2019

(N=241)

January/

February 2020

(N=266)

January/

February 2020

(N=278)

January/

February 2020

(N=277)

January/

February 2020

(N=347)

Baseline October 2019

(N=196)

October 2019

(N=202)

October 2019

(N=199)

February 2020

(N=241)

February 2020

(N=249)

February 2020

(N=234)

February 2020

(N=347)

Endline 1 December

2019/ January

2020

(N = 185)

December

2019 /January

2020

(N= 190)

December

2019/ January

2020

(N= 190)

March, 2020

(N=221)

March 2020

(N=227)

March 2020

(N=229)

March 2020

(N=204)

Endline

Behavioral

April 2020

(N= 57)

April 2020

(N= 48)

April 2020

(N= 51)

April 2020

(N=173)

April 2020

(N=178)

April 2020

(N=163)

April 2020

(N= 98)

Endline 2 May 2020

(N=149)

May 2020

(N=172)

May 2020

(N=161)

Notes Jahangirnagar

University had

a three week

delay.

Jahangirnagar

University had

a three week

delay.

Jahangirnagar

University had

a three week

delay

Endline 2 used

for those

affected by

Covid-19

delay

Endline 2 used

for those

affected by

Covid-19

delay

Endline 2 used

for those

affected by

Covid-19

delay

OUTCOMES OF INTEREST AND MEASUREMENT STRATEGY

We measure participant norms, skills, confidence and competence to act, and willingness to intervene in

defense of a victim using a combination of survey instruments and a web-based stimulus activity.

Summary statistics for all outcomes are presented in table 14 (appendices). Each component of the

measurement strategy (following the pathways articulated in section 4) is discussed in the following

subsections below:

Participants were told that attendance of at least 12 of the 14 sessions (with some variation depending on school

circumstances, and some exceptions made on a case by case basis by the facilitator) was the only requirement for receiving the

certificate. In practice, the vast majority of participants who attended at least 10 sessions were awarded the certificate.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 28

NORMS

We consider two categories of social norms that the tolerance BIT programming was meant to

promote: (1) norms of tolerance and (2) the rejection of violence. These, in turn, were measured as

follows:

NORMS OF TOLERANCE AND INTOLERANCE

We constructed indices for a battery of outcomes measuring changed norms of tolerance and

intolerance. The theme of promoting and protecting norms of tolerance was a key component of the

program. The broad nature of the concept of tolerance implied that we should operationalize it by

measuring changed attitudes regarding a variety of salient social cleavages in Bangladesh. We also

focused on religious tolerance in depth by using an Inquisit-based stimulus activity designed to sensitize

respondents to contemporary religious conflict before measuring attitudes. We divide tolerance and

intolerance measures into two groupings, by modality of measurement.

First, we create aggregated

indices measuring the following, both at baseline and endline:

• Religious Outgroup Disgust: An index of survey items asking the respondent the extent to which

they agree that different religious traditions (omitting their own) are a “corrupting influence within

Bangladesh.” The intention of the program was to promote norms against such disgust.

• Hostile and Benevolent Sexism: Adapted from the Glick and Fisk (1996) Ambivalent Sexism

Inventory Scale.

We distinguish hostile sexism (which is a measure of the extent to which the

respondent perceives women in society as threatening) from benevolent sexism (which is a measure

of the extent to which the respondent believes that women should be protected and cherished

within society). The intention of the program was to promote norms against such sexism.

• Anti-LGBT Disgust: Although the programming did not include formal content on LGBT issues, we

included a measure of attitudes towards LGBT persons in baseline/endline surveys as a single

question adopted from Harek’s (1984) 20 item attitude towards LGBT persons. The intention of the

program was to promote norms against such disgust.

• Anti-Western Attitudes: An index of questions asking the participant the extent to which they

believe that secularism and the West are corrupting influences in Bangladesh. The intention of the

program was to promote norms against such attitudes.

• Distrust of Foreigners: An index of questions asking the participant the extent to which they are

suspicious of foreigners, and the extent to which they believe that foreign customs are changing

We divide the analysis by measurement modality because April 2020 Covid-related shutdowns impacted data collection for

the Inquisit experiments. The experiments were originally planned with an on-campus computer lab option with assistance from

a trained student “tech coordinator.” However, due to Covid-19 shutdowns, they were instead administered entirely remotely

using the Inqusit app. While this proved successful for about half of the sample, many participants could not successfully run the

app on their devices, given that the software required Android versions 6.0 or above at the time. For the statistical analysis, we

have thus separated the Inquisit-based measures from Qualtrics survey measures due to smaller sample sizes.

Glick, Peter, and Susan T. Fiske. "The ambivalent sexism inventory: Differentiating hostile and benevolent sexism." Journal of

Personality and Social Psychology 70.3 (1996): 491.

The curriculum did not explicitly distinguish the technicalities that differentiate hostile sexism from benevolent sexism,

though we believe that examples of both were discussed in sessions.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 29

their own customs too much. The intention of the program was to promote norms against such

distrust.

We further explored the concept of religious tolerance/intolerance by using an endline-only web-based

activity that presented respondents with a vignette designed to sensitize them to contemporary religious

conflict. Respondents were asked to read an excerpt from an April 2020 BBC article documenting anti-

Muslim riots occurring in neighboring India.

100

Following the stimulus, respondent attitudes were

measured (using a Likert agreement scale) for the following concepts, which we collectively dub

“religious group status:”

• Religious Nationalism: A statement that “Bangladesh is a Muslim country above all else” which

measures the extent to which the respondent ties his/her national identity to their religious identity.

The intention of the program was to dampen attachment to such nationalism.

• Religious Segregation Preferences: An index of two items stating that Hindus (Muslims) who live

in non-Hindu (non-Muslim) countries should move to Hindu (Muslim) countries, respectively.

• Religious Collective Narcissism: A statement that in Muslim countries, Muslims should have more

political rights than other groups.

• Religious Perceived Group Victimhood: A statement that Muslims around the world are under

threat.

Together, these measures capture the extent to which participants perceive the world around them in

terms of potential group status loss, characterized by religious identity. We expect that evidence of

program efficacy would be reflected in relatively lower levels of these measures in those receiving

tolerance-focused BIT treatment when compared to those in placebo and no-program control groups,

following exposure to being primed with contemporaneous information about anti-Muslim violence in a

neighboring country.

REJECTION OF VIOLENCE NORM

The second major social norm targeted by the curriculum focused on use of violence.

101

We evaluated

participant attitudes and norms about violence as follows at endline:

• Violence Disapproval: An index of questions drawn from Moskalenko & McCauley’s (2009)

“Radicalism Intentions Scale”.

102

This scale asks the respondent to name a cause that a close friend

of theirs cares deeply about, followed by questions measuring the extent to which they would

approve of the close friend engaging in a range of activities on behalf of that cause. For our

100

At the time, anti-Muslim riots in India had a great amount of salience within Bangladeshi media and society, as they were

occurring concurrently with changes being made to Indian citizenship laws that were widely perceived as discriminatory

towards Indian Muslims. We chose this specific article because we wanted to measure facets of religious separatism/supremacy

among participants (the vast majority of whom identified as Sunni) when exposed to a current events story they’d realistically

encounter over social media.

101

Via endline surveys prior to web-based Inquisit activities.

102

Moskalenko, Sophia, and Clark McCauley. "Measuring political mobilization: The distinction between activism and radicalism."

Terrorism and Political Violence 21.2 (2009): 239-260.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 30

purposes, we used two questions defined by Williams, Horgan, and Evans (2016) as “amped political

extremism,” which focus on the use of violence on behalf of the cause.

103

• Violence Worriedness: This index uses the same “radicalism intentions” scale as that of “violence

disapproval,” but instead asks the respondent how worried they would be about a friend engaging in

the aforementioned violent activities.

• Rejection of VEO Goals: A question measuring the extent to which the respondent believed that

groups committing violent acts sometimes had legitimate goals. We designed this battery of

questions in order to avoid asking the respondent directly about their attitudes towards specific

VEOs, due to safety concerns. Given that the programming was designed to promote and reinforce

norms against violence (without focusing on any particular violent extremist

organization/movement), we expect that evidence of anti-violence norm internalization would be

reflected in rejection of the notions that VEO goals can be legitimate.

104

• VEO Awareness: An index of questions we created to measure the respondent’s views regarding

the perceived popularity and reach of violent extremist organizations within Bangladesh and among

those like themselves. We designed this battery of questions in order to avoid asking the

respondent directly about their attitudes towards specific VEOs, due to safety concerns. We expect

that evidence of an anti-violence norm internalization would be reflected in an increased recognition

that VEOs are present and actively recruiting within Bangladesh, and that they frequently target

those similar to themselves.

105

• Openness to VEO Demobilization: Finally, we probed the extent to which participants

distinguished the use of violence for pursuing extremist goals from the non-violent pursuit of

extremist goals. An index of two questions was constructed measuring the extent to which the

respondent agreed that VEOs should be permitted to compete in the political process if they

renounce the use of violence, and whether it’s okay to agree with the goals of VEOs while

disagreeing with their methods of pursuing the goals. We expect that evidence of anti-violence norm

internalization would be reflected in acceptance of the notion that a VEO which renounced the use

of violence should be permitted to participate in the political process.

103

Williams, Michael J., John G. Horgan, and William P. Evans. "Evaluation of a multi-faceted, U.S. community-based, Muslim-led

CVE program." (2016).

104

Design meetings with Rupantar and consultations with USAID emphasized that asking respondents directly about whether

they supported the goals of VEOs would be unsafe, even with the privacy and confidentiality steps we took. Moreover, doing so

would likely yield untruthful responses due to social desirability bias. Instead we framed these items using indirect statements

such as “the political goals of violent extremist organizations like ISIS are popular within Bangladesh,” or “It is okay to agree

with the political goals of violent extremist organizations without supporting their methods.” Framing the items in this way

carries the downside of less precision in the concept, an issue we discuss further in the results.

105

Design meetings with Rupantar and consultations with USAID emphasized that asking respondents directly about whether

they supported the goals of VEOs would be unsafe, even with the privacy and confidentiality steps we took. Moreover, doing so

would likely yield untruthful responses due to social desirability bias. Instead we framed these items using indirect statements

such as “the political goals of violent extremist organizations like ISIS are popular within Bangladesh,” or “It is okay to agree

with the political goals of violent extremist organizations without supporting their methods.” Framing the items in this way

carries the downside of less precision in the concept, an issue we discuss further in the results.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 31

SKILLS

We measure skills using two sets of outcomes, which gauge (1) participant retention of key concepts

from the tolerance-focused BIT curriculum in the form of a knowledge assessment, and (2) the mastery

of skills specifically required for effective bystander intervention.

First, measured participant retention of key concepts from the curriculum by making use of a 10-

question common knowledge assessment (CKA), which was developed by Management Systems

International (MSI).

106

The CKA consisted of 10 questions which focused on the drivers of peace,

harmony, stability (and instability) in Bangladeshi society, as well as the potential drivers of violent

extremism within Bangladesh. Participants could select as many answers as they felt were applicable.

From the CKA, we developed a “rubric” which awarded higher scores for selected answers that were

consistent with the programming,

107

while penalizing answers that were not consistent with the spirit of

the programming.

108

Specifically, CKA scores were constructed measuring:

• Tolerance CKA: Generated from 6 CKA questions for which tolerance (and related concepts)

were relevant potential answers to questions about peace, harmony and lack thereof in Bangladesh.

• Violent Extremism Drivers CKA: Generated from 2 CKA questions which ask the participant about

the drivers of violent extremism. Higher scores were those that linked intolerance (and related

concepts) to violent extremism.

Second, effective bystander intervention requires mastery of skills to go from noticing a social

emergency to actually acting in order to de-escalate it or otherwise help in a safe manner. In order to

measure the BIT skills in participants, we draw on Burn’s (2009)

109

operationalization of the Latane and

Darley (1970)

110

five barrier situational model of bystander intervention. This battery of questions

measures skill-based impediments to intervention, starting with their likelihood of noticing a problematic

situation in the first place. We adopted this battery of 14 questions to the context of Bangladesh by

focusing each hypothetical statement on situations of religious, and sex-based harassment and

discrimination. These 14 questions were then aggregated into outcome indices as follows:

• Failure to Notice: Two items positing the likelihood that the respondent would be unlikely to

notice perpetrator behaviors at a gathering.

• Failure to Identify an Emergency: Three items positing the likelihood that the respondent would

be unlikely to correctly identify a situation that has a high potential for escalating to violence, despite

having noticed such a situation.

106

MSI was awarded the prime contract for managing the implementation of the Obirodh Project, which included this program,

among others.

107

For example, one question asks “what causes societal conflict?” Participants would score higher on the CKA if they selected

the answer “misunderstanding of different cultures and practices.”

108

For example, one question asks “what makes a society cohesive?” Participants would score lower on the CKA if they

selected the answer “a society with the same culture, religion and clothes.”

109

Burn, Shawn Meghan. "A situational model of sexual assault prevention through bystander intervention." Sex Roles 60.11

(2009): 779-792.

110

Latané, Bibb, Bibb Latané, and John M. Darley. The unresponsive bystander: Why doesn't he help? Appleton-Century-Crofts,

1970.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 32

• Failure to Assume Responsibility: Five items positing the likelihood that the respondent would be

unlikely to assume that they have personal responsibility to act in a given situation, even if it is a

clear emergency.

• Failure due to Skill Deficit: Two items positing the likelihood that the respondent would be

unsure of what to say or do even if they believed they had responsibility to intervene.

• Failure due to Audience Inhibition: Two items positing the likelihood that the respondent would

be afraid to act even if they knew how to, due to fear of judgement from others or that they would

not receive help from others.

CONFIDENCE AND SELF-ASSESSED COMPENTENCE TO ACT

Beyond the internalization of norms and skills, effective bystander intervention requires a sense of self-

efficacy, i.e., the self-evaluated sense of confidence and competence to act. Thus in addition to

internalized norms and skills associated with the programming, we measure participant evaluations of

their own sense of agency by constructing three indices of questions adapted from Banyard et al.’s

(2014)

111

“readiness to help” scale for measuring bystander behaviors related to sexual assault on

university campuses. This set of items asks participants to evaluate their willingness, readiness, and ability

to act in order to address a variety of hypotheticals associated with sexual assault. Items include, for

example: “I think I can do something about sexual assault,” and “There is not much need for me to think

about sexual assault.” We adapted 9 of these items for each of three topic areas relevant to the

Bangladeshi context:

• Religious Intolerance: Items describing confidence/competence to act under scenarios of religious

intolerance being expressed on campus.

• Violence on Campus: items describing confidence/competence to act under scenarios of violence

on campus. We chose to leave the nature of hypothetical campus violence vague because much of

campus violence in Bangladesh is political violence carried out by student political groups.

• Extremism: items describing confidence/competence to act under scenarios of violence on campus.

“Extremism” and “violent extremism” are viewed interchangeably in Bangladesh and have the

same Bangla translation, unless explicitly differentiated as “non-violent extremism.”

In addition to these self-evaluations of confidence/competence, we also included a battery of items

evaluating the respondents’ perceptions that their friends would act under a variety of context-relevant

hypotheticals. These 11 items were drawn from Banyard et al.’s (2014) Perceptions of Peer Helping

Index.

112

A participant’s perceptions about their friends/social circle’s ability and willingness to help in a

social emergency reflects their own sense of confidence to affect change in such a situation. The

participant’s perceptions about their friend’s willingness to help is a direct indicator of perceived social

norms regarding helping/intervention behavior.

113

111

Banyard, V. L., et al. "How do we know if it works? Defining measurable outcomes in bystander-focused violence

prevention." Psychology of Violence 4.1 (2014): 101-15.

112

Ibid.

113

This is distinct from social norms of tolerance.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 33

WILLINGNESS TO INTERVENE

Finally, we evaluated effective bystander intervention capacity among participants by presenting them

with four vignettes, each describing different perpetrator/victim interactions focused on the following

topics:

• Religious Discrimination: In this vignette, the participant is asked to put themselves in the place of

a person witnessing a perpetrator who uses an anti-Hindu slur.

• Sexual Harassment (“Eve-teasing”): In this vignette, the participant is asked to put themselves in

the place of a person witnessing a perpetrator who “eve-teases” a woman walking by.

• Anti-LGBT Bullying: In this vignette, the participant is asked to put themselves in the place of a

person witnessing a perpetrator who bullies a student who is rumored to be homosexual.

• Radicalization: In this vignette, the participant is asked to put themselves in the place of someone

who witnesses a friend become angry at a news story, discuss having met a new group of friends

online who plan to take action for a cause, and mentions that they are “willing to fight, and even die,

to make a difference .”

114

For each of these four vignettes, participants were presented with an expanded list of potential

responses and asked their likelihood of choosing that response for the scenario. Some of the responses

offered included bystander intervention training strategies (which the participants were taught as part of

the training), whereas others were considered “do nothing” responses (i.e., “I would wait and see what

happens”), and others are considered “join perpetrator” responses (i.e., “I would encourage him (the

radicalized person) if it were for the right reasons.” From these menus of responses, we constructed 12

outcome variables (three for each of the four topic areas), as follows:

• BIT Strategies Used: An index of respondent responses to their likelihood of using one or more

BIT strategies in response to the vignette. So, for example, respondents who indicated they were

highly likely to “attempt to create a distraction” or “delay your response and check in later with the

victim” ranked higher on this index.

• No Inaction: A reversed index of responses to the likelihood of a passive response/inaction in

response to the scenario. Respondents indicating they were highly unlikely to “do nothing” or

“watch and see what happens” scored higher on this index.

• Not Joining Perpetrator: Finally, respondents who indicated that they were highly unlikely to join

the perpetrator in the given vignette scored higher on this metric (which was also reversed).

114

We verified that this scenario would be understood as a case of likely radicalization through a small pilot study using

Mechanical Turk (in which respondents overwhelmingly described it as such a scenario in free responses). Rupantar also

confirmed that it was almost certainly likely to be understood this way.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 34

PEER LEADERS VS TRADITIONAL AUTHORITY FIGURES

IDENTIFICATION WITH FACILITATOR

In addition to comparing the efficacy of peer leaders and traditional authority figures on the previously

discussed outcomes, participants were asked to respond to a battery of items rating their facilitator on a

variety of personal traits, in order to verify the extent to which the “peer leader” and “traditional

authority” facilitators were perceived as we would expect them to be. We aggregated these trait-related

items into the following:

• Relatability: An index of four items rating the extent to which the facilitator is relatable, including

statements such as “my facilitator has walked a similar path as me” and “my facilitator is just like

me.”

• Experience: An index of six items rating the extent to which the facilitator is experienced, including

statements such as “my facilitator is knowledgeable,” and “my facilitator has seniority.”

• Youthfulness: An index of two items rating the extent to which the facilitator is youthful, including

the statements “my facilitator is youthful” and “my facilitator is enthusiastic.”

• Approachability: An index of four items rating the extent to which the facilitator is approachable,

including statements such as “I would feel comfortable telling my facilitator about a personal

problem,” and “I would feel comfortable asking my facilitator to explain something I don’t

understand.”

IDENTIFICATION WITH SOCIAL ENVIRONMENT OF PROGRAM

A second set of facilitator-driven outcomes we evaluate are participant perceptions of the social

environment of the programming. Specifically, the extent to which the participant feels socially

connected to their classmates (which is moderated by the facilitator) is likely to affect their identification

with program goals. We measure participant social experiences using two variables: the extent to which

they feel that they made new friends from the programming, and the extent to which they would

recommend the programming to their friends. Table 11 depicts results from the omnibus “mean effects”

test for each social outcome, followed by item-level effects.

BASELINE CHARACTERISTICS COVARIATE BALANCE

Table 3 displays covariate balance across the four treatment contrasts. In addition to basic

demographics, we compare participants on a variety of baseline psychosocial variables and attendance

rates (attrition). We aggregate conceptually similar, correlated measures into indices.

115

Overall, the

contrast between tolerance BIT program recipients and placebo (either placebo or tolerance BIT) was

well-balanced on demographic and psychosocial baseline variables (Table 3 Col 8). An omnibus test of

differences in baseline demographic and psychosocial covariates suggests no systematic differences.

115

For example, we combine Likert-scale measures of depression, emotional stability, economic optimism, and recent loss into

an index we label “mental health.” We use a similar approach for aggregating anger, hostility, verbal aggression, and hostile

attribution bias into an index we label “anger.”

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 35

When comparing peer and traditional authority groups within the BIT tolerance programming (Table 3,

Col 9), we found that participants in traditional authority-led groups were more likely to report prior

familiarity with their facilitator than those enrolled in the peer leaders group. This was not surprising,

given that the participating students may have had classes before with the professors serving as

facilitators (or know of them through their departments). We also found that those enrolled in

authoritative facilitator-led groups were more likely to report good relationships with their family

members, less likely to report having had negative prior interactions with those from different religious

and/or ideological backgrounds, and more likely to report prior exposure to classes/programming that

develops critical thinking skills. We include family relations quality, prior negative interactions with out-

groups, and prior critical thinking skills class/program exposure in specifications contrasting peer and

authoritative influencer groups.

116

Those who opted to participate in a wave 2 no-program control were more likely to be older, and male

than those enrolled in the pooled tolerance BIT programming sample (Table 3 Col 6). Other

demographics and psychosocial measures were not found to be unbalanced. No-program control

participants were incentivized with a small gift card, and they included a combination of those from the

randomized programming list who did not participate in the programming and some who were recruited

later via flyers. We believe that no-program control participants may have been unbalanced on age and

gender demographics due to covid-19 related shutdowns, which necessitated participants’ connections

from their home villages via their personal devices and/or laptops. This problem was mitigated to some

extent by the provision of data packages by NORC at the University of Chicago for remote

connectivity. However, we suspect that older male students are more likely to have personal devices

and/or laptops (or access to them) in the first place when no longer on their university campuses. We

control for age and gender in specifications that include the no-program control participants.

Regarding attrition, attendance levels (total classes attended), dropout rates (those who started but did

not finish the program), and “no-show” rates (those who completed a baseline but never started the

program and were replaced) were not found to be correlated with treatment assignment. We also

conducted baseline covariate balance tests comparing finishers (defined as those who completed at least

10 of the 14 sessions), drop-outs (those who completed between 1-9 sessions), and no-shows (those

who were admitted and completed intake data, but never showed up for the first session and were

replaced). Of the various demographic and psychosocial background traits we measured, we only found

differences in reported social media use; those who never showed up and those who dropped out

reported greater baseline social media use than those who finished the program.

Greater levels of social media use among drop-outs was unexpected; if anything, we would have

expected social media use to be a proxy for digital connectivity (which would be associated with having

finished, given that many wave 2 students finished programming online). It is possible that those

reporting higher levels of social media use were also more likely to find other (competing) activities,

given that Facebook is commonly used to organize and advertise extracurriculars.

Baseline covariate balance by attrition status is depicted in Table 15 (appendices).

116

No-program control participants were not assigned questions related to the programming/facilitators, given that they did

not participate in programming.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 36

Table 3: Pre-Treatment Covariate Balance

117

Baseline

Covariate

NPC

(204)

Mean

[SE]

Placebo

(406)

Mean

[SE]

Auth

(418)

Mean

[SE]

Peer (422)

Mean

[SE]

Diff

(t-test)

(NPC vs

BIT All)

Diff

t-test)

(NPC +

Placebo vs

BIT All

Diff

(t-test)

(Placebo

vs BIT

All)

Diff

(t-test)

(BIT Auth

vs BIT

Peer)

(Omnibus Test) -0.30 [0.06]

<0.01

[0.02]

< 0.01

[0.02]

<0.01

[0.02]

<0.01*** 0.77 0.81 -0.84

Age

0.24

[0.09]

[0.05]

-0.06

[0.05]

-0.06

[0.05]

0.29*** 0.12*** -0.01 0.01

Male

0.71

[0.03]

0.57

[0.03]

0.56

[0.03]

0.61

[0.02]

0.13*** 0.04 0.03 -0.05

Sunni

0.78

[0.03]

0.79

[0.02]

0.78

[0.02]

0.75

[0.02]

0.02 0.03 0.05 0.03

Econ. Welfare

-0.12

[0.08]

0.06

[0.05]

0.03

[0.05]

0.01

[0.05]

-0.13 -0.01 0.05 0.01

Anger

0.05

[0.08]

0.01

[0.05]

-0.05

[0.05]

-0.02

[0.05]

0.1 0.07 0.05 -0.05

Grit

-0.13

[0.07]

0.08

[0.05]

0.03

[0.05]

-0.03

[0.05]

-0.15 -0.02 -0.05 0.04

Deliberativeness

-0.08

[0.08]

-0.01

[0.05]

0.09

[0.05]

0.05

[0.05]

-0.14 -0.08 -0.04 0.08

Mental Health

0.04

[0.07]

0.02

[0.05]

0.08

[0.05]

0.02

[0.05]

0.04 -0.01 -0.06 0.06

Pro-sociality

0.04

[0.07]

-0.02

[0.05]

-0.07

[0.05]

-0.03

[0.05]

0.1 -0.01 -0.01 0.02

Social Media Use

-0.07

[0.07]

-0.02

[0.05]

-0.07

[0.05]

-0.04 -0.02 -0.03 -0.07

Family Relations

-0.09

[0.08]

0.01

[0.05]

0.13

[0.04]

-0.03

[0.05]

-0.12 -0.06 0.01 0.17***

Family Overseas

3.18

[0.16]

3.46

[0.11]

3.41

[0.11]

3.49

[0.11]

-0.27 -0.09 0.01 -0.07

Religiosity

(Reversed)

0.08

[0.07]

-0.01

[0.05]

-0.02

[0.05]

-0.01

[0.05]

0.12 0.05 -0.08 0.01

Diversity of

Friends

-0.06

[0.07]

-0.07

[0.05]

0.03

[0.05]

-0.06 -0.07* -0.08 0.06

117

Robust standard errors in brackets. The value displayed for t-tests are the differences in the means across the groups. All

estimation regressions include school and wave fixed effects. ***, **, and * indicate significance at the 1, 5, and 10 percent

critical level.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 37

Baseline

Covariate

NPC

(204)

Mean

[SE]

Placebo

(406)

Mean

[SE]

Auth

(418)

Mean

[SE]

Peer (422)

Mean

[SE]

Diff

(t-test)

(NPC vs

BIT All)

Diff

t-test)

(NPC +

Placebo vs

BIT All

Diff

(t-test)

(Placebo

vs BIT

All)

Diff

(t-test)

(BIT Auth

vs BIT

Peer)

Negative Out-

Group

Interactions

-3.96

[0.17]

-4.14

[0.12]

-4.25

[0.12]

-3.93

[0.12]

0.12* -0.01 0.03 -0.36**

Critical Thinking

Exposure

[0.07]

-0.02

[0.05]

0.02

[0.05]

-0.12

[0.05]

-0.09* -0.01 -0.09 0.14**

Prior

Programming

Exposure

-0.08

[0.05]

0.01

[0.05]

0.05

[0.05]

NA NA 0.12* -0.07

Prior Facilitator

Familiarity

0.09

[0.06]

0.08

[0.06]

-0.14

[0.04]

NA NA 0.04 0.22***

Finished

Program

0.75

[0.02]

0.78

[0.02]

0.80

[0.02]

NA NA -0.03 -0.02

Sessions

Attended

8.84

[0.22]

9.04

[0.21]

9.00

[0.22]

NA NA -0.18 0.04

No-Show NA

0.18

[0.02]

0.16

[0.02]

0.19

[0.02]

NA NA 0.17 -0.02

6. DATA ANALYSIS

Our data analysis aims to addressing the following research questions:

RQ1.

Does the P/CVE Bystander Intervention Training (BIT) Curriculum increase participants’

motivation and capacity to challenge individuals taking actions that reflect violent extremism? In

particular, does it cause participants to:

(1) Internalize norms against extremism,

(2) Possess skills for recognizing and knowing how to safely challenge behaviors and

expressions of attitudes linked to violent extremism,

(3) Possess confidence in their ability to challenge such behaviors and attitudes, and

(4) Put all of these things together effectively to challenge behaviors and expressions of

attitudes linked to violent extremism?

RQ2. Is the program more effectively delivered by trained peer leaders or by traditional

authority figures?

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 38

Following our theory of change, we estimate effects on items measuring the following:

• Internalization of norms, including the following:

− Intolerance (we expect these to be reduced)

− Rejecting violence (we expect these to be increased)

• Skill- and knowledge-based barriers to challenging violent extremism antecedents (we expect these

to be decreased)

• Confidence and self-assessed competence to challenge violent extremism antecedents (we expect

these to be increased)

• Willingness to Intervene in scenarios modeling violent extremism antecedents (we expect these to

be increased)

• Identification with facilitator (we expect this to vary dependent on peer versus traditional authority

facilitator).

We discuss the specific measures in the “outcomes of interest” subsections below that describe our

results.

For RQ1 regarding the effect of the program on norms, skills, confidence, and action against violent

extremism, we report estimates from regression models specified as follows:

Endline Outcome = a + b*BIT Curriculum + c*Baseline Outcome + University FE

+ Wave FE + Unbalanced Covariates + e (1)

where a is an intercept term that measures mean outcomes in the absence of the BIT intervention, b

measures the effect of the BIT intervention, c measures how the baseline measures of the outcomes

correlate with endline measures, the University FE and Wave FE terms refer to indicator variables that

control for variation at the level of universities and data collection waves, and the Unbalanced Covariates

are control variables determined on the basis of analysis of pre-treatment covariate data, and e is an

error term that captures other idiosyncratic variation from individual to individual. We estimate the

regression using weighted least squares, weighting by the inverse probability of treatment assignment.

We use heteroscedasticity robust standard errors, given that the treatment assignment was at the level

of individuals.

118

We report two types of effects for RQ1. The first effect is based on the contrast between tolerance BIT

program participants versus those in the no-program control condition. We do this by estimating

equation (1) on a sample that includes program participants and those who were not engaged in any

program at all, and excludes those who received the placebo curriculum. This estimates the gross

impact of the program, including the effect of the BIT curriculum per se, as well as any effects

attributable to getting people together in a novel social situation to think critically. The second effect

118

As a technique of accounting for class cohort spillovers, we also ran specifications that clustered standard errors on

facilitator class cohorts. Results were found to be negligibly different from those reported in the results section below.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 39

isolates the effect of receiving the BIT curriculum as compared to receiving the placebo curriculum. This

effectively controls for the aforementioned social interaction effects, which would be operative in both

the BIT and placebo curricula (therefore enabling comparison of differences between the tolerance BIT

and placebo). To do this, we estimate specification (1) on a sample that includes those in the BIT

curriculum and those in the placebo curriculum, excluding those who were not engaged in any program

at all. This estimates the added value of the curriculum per se, above and beyond the effect of the novel

social interactions and critical thinking exercises that were also included in the placebo curriculum.

119

For RQ2, which measures the effect of having a peer versus traditional authority facilitator, we use the

following specification:

Endline Outcome = a + b*Peer Facilitator + c*Baseline Outcome + University FE

+ Wave FE + Unbalanced Covariates + e (2)

where, in this case b is the effect of having a Peer Facilitator instead of a Traditional Authority facilitator,

and all of the other terms have the same meaning as in equation (1). Estimation again uses weighted least

squares with heteroscedasticity robust standard errors. This effect characterizes whether peer

facilitators generate any different effects with the tolerance curriculum as compared to traditional

authorities.

We use a hierarchical testing and statistical significance strategy to first evaluate overall effects on sets of

items, and then to evaluate item-level effects that are adjusted to account for the fact that we are doing

many tests (i.e., to address the “multiple comparisons problem”). As discussed above, we have sets of

items measuring our concepts. For example, for norms of intolerance, we have six items: a Religious

Outgroup Disgust index, Hostile Sexism index, Benevolent Sexism index, Anti-LGBT Disgust index,

Anti-Western Attitudes index, and Distrust of Foreigners index. To test overall effects, we use an

omnibus “mean effect” test constructed as the average of z-score standardized effects for each item.

120

We implement this using the mseffect command in Stata. Then, within each group of items, we report

item-level effects and with multiple comparisons false discovery rate (FDR) p-value adjustment using the

mhtreg command in Stata.

121

Methodologists refer to such adjusted item-level p values as “q values.” The

item-level results allow us to assess which items may be driving the overall omnibus effect, and the q

values allow us to test for statistical significance in a way that accounts for the multiple testing problem.

In our discussion, we take effects with p or q values less than 0.05 are considered to be statistically

significant, while when we have p or q values between 0.05 and 0.10, we refer to the finding as

“suggestive.”

119

As a technicality, we estimate this curriculum effect using two approaches: one includes the no-program control group so

only to get more precise estimates on the control variables, and the other excludes them. Both specifications target the same

effect, and yield essentially the same results.

120

This follows the approach of Kling, J.R., Liebman, J.B. and Katz, L.F. (2007), Experimental Analysis of Neighborhood Effects.

Econometrica, 75: 83-119. For an explanation of hierarchical testing strategies for program evaluation, see also Casey, K.,

Glennerster, R., & Miguel, E. (2012). Reshaping institutions: Evidence on aid impacts using a preanalysis plan. The Quarterly

Journal of Economics, 127(4), 1755-1812.

121

This follows the approach presented in Michael L. Anderson (2008) Multiple Inference and Gender Differences in the Effects

of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects, Journal of the American

Statistical Association, 103:1481-1495.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 40

7. RESULTS

In this section, we report results for the effects of the tolerance BIT programming, followed by the

effects of peer leaders (compared to traditional authority figures) in delivering the programming. For

both components of the field experiment, we present results according to the causal chain specified in

figure 1 (above). Results are depicted as coefficient plots for all outcomes in this section, whereas tables

reporting coefficients, standard errors, p-values and FDR (q-values) are presented in the appendices, and

referenced where appropriate. As discussed in the research design, outcomes are generally constructed

as aggregated additive indices of Likert-scale survey and post-activity measures, which were then

standardized using pooled group mean and standard deviation. Therefore, all effects that we report

below are in pooled standard deviation units. Summary statistics for each outcome category and scaling

are also presented in the appendices. We begin by reporting the program’s estimated effects on social

norms of tolerance and rejection of violence

NORMS

Per the research design, we consider two categories of social norms that the tolerance BIT

programming was meant to promote: (1) norms of tolerance and (2) the rejection of violence. I

NORMS OF TOLERANCE AND INTOLERANCE

Figures 2-4 depict coefficient plots for the omnibus “mean effects” test for each category of outcomes

(two focusing on norms of tolerance and one on rejection of violence), followed by item-level effects for

each outcome index. All specifications include unbalanced covariates and wave/school fixed effects.

Tables 4-6 (appendices) report coefficients, standard errors, p-values and q-values.

In terms of gross effects of programming, we find that intolerance was reduced by about an eighth of a

standard deviation (ca. 0.12 sd; Table 4, column 1); this represents a modest although statistically

significant effect. Changes were most evident in benevolent sexism, with changes also apparent in anti-

LGBT, anti-Western and anti-foreigner sentiment. For benevolent sexism, it is possible that the nature

of the coed interactions themselves (which were in place under both the BIT and placebo curricula) had

an effect on its own, independent of the content of the curriculum. Changes in anti-LGBT sentiment

were interesting, given the aforementioned discussion about the decision not to include curriculum

content focusing on LGBT issues in any program elements.

About half or a little more of the summary index effect appears attributable to the tolerance BIT

curriculum per se—0.07 standard deviation (Table 4, column 4). The extent to which the tolerance BIT

curriculum mattered varies by item. The curriculum seemed to be associated with decreases in anti-

religious outgroup sentiments and anti-Western attitudes, with suggestive evidence pointing to declines

in hostile sexism.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 41

Figure 2: Effects of Programming and Tolerance BIT Curriculum on

Norms of Tolerance

Figure 3: Effects of Programming and Tolerance BIT Curriculum on

Norms of Religious Group Status Measures

For the web-based stimulus activity on religious status, we find an omnibus effect of similar magnitude as

the previous results (Table 5, column 1) and consistent with our expectations. Because of the smaller

sample size, the differences are not statistically significant for the items. However, the curriculum was

associated with a decline in these metrics via the omnibus, likely reflecting greater power from the

additional placebo participants in the analysis.

122

Note that the smaller sample size is due to the fact that only about half of the sample was able to successfully complete the

activities, following Covid-19 shutdowns.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 42

Figure 4: Effects of Programming and Tolerance BIT Curriculum on

Rejection of Violence and VEOs

NORMS AGAINST VIOLENCE

Regarding the rejection of violence and violent extremist organizations, we find that the summary effects

(Table 6, columns 1 and 4) are small when examining gross effects of programming. Curriculum effects

seem to be more pronounced; the tolerance curriculum was associated with about one sixth of an

increase in rejection of violence, driven specifically by increased disapproval and worry about a

potentially violent friend.

Findings for gross effects of programming appear driven by the fact that we have positive effects for the

items pertaining to concern for/disapproval of friends’ involvement in violence, but then an unexpected

negative effect for rejection of VEO goals among those in all programming; As noted, this item is

measuring the extent to which the goals of VEOs can be legitimate despite the fact that VEOs use

violence to pursue them. The findings here make this distinction important: goals despite violence,

versus violence as a tactic per se. It is therefore plausible (though this is speculative) that prompting

subjects to thinking critically (whether in focusing on gender and religious tolerance in the BIT

curriculum or on more neutral subject matter in the placebo) would actually make people more likely to

distinguish between goals, which can be legitimate on their own, and means of pursuing those goals, in

this case, through violence. This finding is also important with regard to the concern about social

desirability bias. If subjects were merely parroting what they took to be the socially desirable response,

rather than responding in a way that reflect genuine reflection and judgment, we certainly would not

expect subjects to evaluate VEO goals in this way. This effect suggests that subjects were prompted

to apply a more critical mindset, rather than mindlessly parroting norms of tolerance. Further support

for this interpretation comes from the findings with respect to “openness to VEO demobilization”,

which specifically examine the participant’s willingness to accept those connected to VEOs if they

renounce violence. The programming increased participants’ rejection of the use of violence among their

friends/peer groups, but it also prompted subjects to distinguish goals from means and to be less

reflexive in rejecting the possibility that those connected to VEOs are irredeemable because of part

connection to violence.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 43

Overall, we observe improved tolerance among participants. We can attribute about half of the effect to

the curriculum and the rest to the novel social interactions and general prompts to think critically.

Program effects on norms related to violence and VEOs suggest improved norms against violence,

though with some nuance regarding subjects’ increased willingness to distinguish goals from actions, and

to be accepting of past VEO-affiliates if they renounce violence. Fortunately, our evaluation does not

stop at evaluating norms and attitudes. What matters ultimately is how subjects act on these new norms

and attitudes. We show encouraging results on that score below, after presenting results on skills and

confidence.

SKILLS

Figures 5 and 6 (below) depict results from the omnibus “mean effects” test for all skills outcomes

followed by item-level effects for each outcome index. All specifications include unbalanced covariates

and wave/school fixed effects. Tables 7 and 8 (appendices) report coefficients, standard errors, p-values

and q-values.

Figure 5: Effects of Tolerance BIT Curriculum on Common Knowledge

Assessment Scores

123

Compared to placebo, those enrolled in the tolerance BIT programming scored about one eighth of a

standard deviation higher on CKA items measuring retention of tolerance-related programming

concepts, and about one fifth of a standard deviation higher on CKA items measuring retention of the

drivers of violent extremism. While it is perhaps unsurprising that those enrolled in a tolerance-focused

program would score higher on tolerance-related knowledge assessments than placebo, it is

nevertheless an important indicator that the participants absorbed and retained the curriculum content.

123

Gross effects of programming cannot be distinguished from curriculum effects because the CKA was not used for no-

program control participants. Figure 5 reports those enrolled in tolerance BIT programming versus placebo only.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 44

Figure 6: Effects of Programming and Tolerance BIT Curriculum on Internalized

BIT Skills

Relative to no program control, we observe strong effects across the board on exercise of skills and

judgment in scenarios of aggression (Table 7, column 1). The curriculum per se (Table 7, column 4)

accounts for about a third of the effect. The tolerance BIT curriculum seemed to be associated with

significant improvements over placebo with respect to training participants to assume personal

responsibility when encountering such scenarios. There is suggestive evidence that the curriculum also

successfully improved participants’ ability to notice and identify an emergency situation, however with a

high q value (0.42). Regarding the impact of the placebo (and the relatively weaker value added of the

tolerance BIT program), it is possible that having participants reflect on their civic responsibility, and

creating group interactions on that basis, can be efficacious in imparting some of the skills required for

effective bystander intervention. Indeed, skills such as noticing aggression, or recognizing and

overcoming audience inhibition, are potentially both improved by any type of programming that asks

participants to engage in prosocial solutions to societal problems. The concentrated attention to BIT

and norms per se therefore may not improve these particular skills much further, although it does not

hurt either.

CONFIDENCE AND SELF-ASSESSED COMPETENCE TO ACT

Figure 7 (below) depicts results from the omnibus “mean effects” test for all confidence/competence

outcomes followed by item-level effects for each outcome index. All specifications include unbalanced

covariates and wave/school fixed effects. Table 9 (appendices) report coefficients, standard errors, p-

values and q-values.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 45

Figure 7: Effects of Programming and Tolerance BIT Curriculum on

Confidence Act

Regarding confidence to act, we found that the tolerance BIT curriculum (figure 7, plot 2) appears to

matter a lot. Program participants (Figure 7, plot 1) appeared to exhibit more confidence to act across

the board when presented with scenarios of religious intolerance, scenarios of violence, and scenarios of

extremism. More than half of these effects are accounted for by the tolerance BIT curriculum (appendix

Table 9, column 4), which presumably is attributable to having participants role play and practice the

skills they learned as a part of the training. Overall, these effects regarding confidence/competence to

act highlight the potential importance of more focused training in prompting participants to progress

from judging a situation (which seems easy enough to do) to seeing themselves as having personal agency

and efficacy in making a difference in the outcome of that situation. It is at this stage in the causal chain

that the BIT curriculum per se starts to matter a lot.

WILLINGNESS TO INTERVENE

Figure 8 (below) depicts results from the omnibus “mean effects” test for all willingness to intervene

outcomes, followed by item-level effects for each outcome index. This is ultimately what the

intervention was intended to inspire. All specifications include unbalanced covariates and wave/school

fixed effects. Table 10 (appendices) report coefficients, standard errors, p-values and q-values. The

results are promising.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 46

Figure 8: Effects of Programming and Tolerance BIT Curriculum on Willingness

to Intervene

We see that from the gross effects (Figure 8 graph 1) and then from the test of the tolerance BIT

curriculum against the placebo (Figure 8, graph 2) that the BIT curriculum matters substantially: the

effects on prior parts of the causal chain carry through to ultimate willingness to intervene. Notably, we

observe effects on two of the three religious discrimination items, two of the three anti-LGBT items,

and one of the three eve-teasing and radicalization items, respectively. Participants were more likely to

select one or more BIT strategies in order to handle scenarios of religious discrimination, sexual

harassment (eve teasing), LGBT harassment, and probable extremist radicalization. The tolerance BIT

programming (Figure 8, graph 2 and appendix Table 10, column 4) accounted for about half of these

effects for three of the four topic areas (the exception being anti-LGBT bullying). Measures of whether

the participant might “do nothing” or “join the perpetrator” in the aforementioned scenarios yielded

less-clear evidence of an effect, although the finding that respondents were less likely to join a

perpetrator in anti-LGBT bullying potentially reflects participant reconciliation between prevailing

societal norms regarding homosexuality and those targeted by the programming. For some participants

holding conservative social views on the subject, the choice to not join a perpetrator in such activities

potentially suggests a reevaluation of how they choose to express those views.

The findings for anti-LGBT bullying might also be suggestive that item floor effects account for a lack of

movement in the other items. (A substantial majority of respondents disagreed moderately or strongly

with the proposition that they would join a perpetrator in these situations, whereas responses were

less-clearly distributed for the LGBT question). For the other items, this is perhaps not surprising, given

that participants who joined and then completed a program of this nature were already likely to exhibit

prosocial characteristics.

Insofar as willingness to intervene is ultimately what matters, these effects are promising. The expected

effects on norms and skills appear to translate into behavioral propensities that create a climate that

promotes tolerance and challenges hateful speech and actions that form the basis of violent extremism.

These effects on behavioral tendencies stand despite the unexpected effect on subjects’ willingness to

distinguish, and accept, the goals of VEOs despite the means that they use to pursue them.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 47

PEER LEADERS VS TRADITIONAL AUTHORITY FIGURES

We now turn to the results regarding the effectiveness of peer leaders as compared to traditional

authority figures, which in our case were university professors. We first consider whether participants

evaluated or identified with peer leaders differently than traditional authority figures. We then examine

the implications for whether peer or traditional leaders serve the goals of the program better.

IDENTIFICATION WITH FACILITATOR

Figure 9 (below) depicts effects for each outcome index. For these specifications, we do not present an

omnibus “mean effects” test, because each of the traits we evaluate are conceptually distinct, and in

some cases are in contrast (i.e., youthfulness versus experience). All specifications include unbalanced

covariates and wave/school fixed effects. Table 11 (appendices) reports coefficients, standard errors, p-

values and q-values.

Figure 9: Participant Identification with Peer Leaders, compared to

Traditional Authority Figures, among Those Enrolled in Tolerance BIT Program

Regarding identification with their facilitator, Table 11 shows that respondents had mixed assessments.

Traditional authority figures were significantly more likely to be seen as experienced- this was not

surprising, given that traditional authorities were older and had higher degrees/job ranks than peer

leaders. Also, not surprisingly, we find evidence that peer leaders were more likely to be seen as

youthful, though with the caveat of a fairly high q value (0.16). However, perhaps unexpectedly,

traditional authority figures were perceived by participants to be more approachable than were peer

leaders. This could potentially be due to clearer institutional relationships between participants and

traditional authority figures (i.e., that of students and professors), whereas such relationships between

participants and their older peers are potentially more nebulous. Likewise, peer leaders themselves

might have felt less confident leading training sessions given their relative lack of teaching experience,

which may have been perceived by participants as an impediment to approachability.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 48

IDENTIFICATION WITH SOCIAL ENVIRONMENT OF PROGRAM

Figure 10 (below) depicts results from the omnibus “mean effects” test for each social outcome,

followed by item-level effects. Table 12 (appendices) reports coefficients, standard errors, p-values and

q-values.

Figure 10: Participant Identification with Social Environment of Programming,

comparing Peer Leaders to Traditional Authorities among Those Enrolled in

Tolerance BIT Program

Figure 10 shows that we do not find clear differences between peer leaders and traditional authority

figures with respect to perceptions of the social environment created by the program. Although

participants with peer leader facilitators were slightly less likely to report making new friends in the

program, and slightly less likely to recommend the program to their other friends, differences were

small and not statistically significant.

We next examine the outcomes of participant identification with programming goals using two items:

the extent to which participants learned about tolerance, and the extent to which participants learned

about bystander intervention training. Figure 11 depicts results from the omnibus “mean effects” test for

each programmatic outcome, followed by item-level effects. Table 13 (appendices) report coefficients,

standard errors, p-values and q-values.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 49

Figure 11: Participant Identification with Programming Goals, Comparing Peer

Leaders to Traditional Authorities, among Those Enrolled in Tolerance BIT

Program

From Table 13 coefficients (column 1), we see that participant self-assessments of the extent to which

they learned key elements of the tolerance BIT curriculum offer some evidence that traditional authority

figures may have been more effective than peer leaders. Namely, those enrolled in peer-led sessions

were less likely to agree with items positing that they learned about BIT strategies.

EFFECT OF PEER LEADERS ON INTERVENTION-RELATED OUTCOMES OF INTEREST

Finally, we compare the efficacy of peer leaders with traditional authority figures with respect to the

other outcomes of interest that we evaluate and present in figures two through eight (above): program

norms (tolerance and rejection of violence), program skills, confidence/competence to act, and effective

use of BIT. For brevity, Figure 12 (below) depicts results from the omnibus “mean effects” test for all

intervention pathway outcomes. For clarity, we print the direction that each relationship between peer

leaders and outcomes would be expected to show if peer leaders were more effective than traditional

authorities.

Regarding the outcomes of interest for the broader program, Figure 12 (below) shows that we do not

find strong differences between using peer leaders to lead programming when compared to traditional

authority figures for the context for most outcomes. To the extent that we do, students of traditional

facilitators seem to do better with respect to anti-violence norms, and with a few component items of

other outcome categories. For brevity, we note these as follows, per each family of outcomes with p-

values and q-values reported in footnotes:

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 50

Figure 12: Impact of Peer Leaders, compared to Traditional Authority Figures, on

Outcomes of Interest among Those Enrolled in Tolerance BIT Program

• Norms: Traditional authority-led participants had anti-foreigner sentiment that was about one sixth of

a standard deviation lower,

124

and participant worry about a potentially violent friend was about one-sixth

of a standard deviation higher,

125

than peer-led participants.

• Skills: Traditional authority-led participants were about one sixth of a standard deviation less likely

to face impediments to intervention rooted in skill deficits than peer-lead participants.

126

• Effective Intervention: Traditional authority-led participants were about one fifth of a standard

deviation less likely to respond passively to an “eve-teasing” scenario.

127

It is possible that these scattered findings in favor of traditional authority figures are consistent with

narratives that those who have prior experience teaching may be better suited for such positions;

however, overall differences between the two groups were not sufficiently clear to conclude that one

provided clearly superior training for those enrolled in the tolerance BIT program.

PLACEBO OUTCOMES: DO NORMS AND CONFIDENCE TO ACT CHANGE WITH RESPECT

TO UNRELATED TOPICS

Finally, we make use of respondent answers to several placebo questions in order to interrogate the

extent to which participants may have simply attempted to provide survey responses reflecting positive

outcomes for the youth social leadership programming that they participated in. Specifically, we

examined whether tolerance BIT program participants exhibited (1) changed norms and (2) changed

levels of confidence to act on issues related to the environment and public health. If tolerance BIT

participants were simply attempting to answer our survey questions in ways that reflected “good

124

p = 0.01; q = 0.07

125

P = 0.03; q = 0.13

126

P = 0.03; q = 0.14

127

p < 0.01; q = 0.16

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 51

leadership,” or if they were attempting to provide us with “good survey answers,” then we would

expect to also observe changed norms and increased confidence to intervene with respect to questions

focusing on “civic virtues” that are completely unrelated to out-group tolerance and violence.

Regarding unrelated norms, an index was created from three questions measuring participant agreement

on items related to (1) punishments for littering, (2) banning smoking from indoor places, and (3)

whether “fast food” was making Bangladeshis sick. Regarding confidence to act, we adapted Banyard et

al (2014)’s “readiness to help” index to a battery of questions related to environmental action, which

appeared in addition to those focusing on tolerance and violence.

128

Using the same analysis framework

articulated in section 6 (above), we test whether gross effects of programming and tolerance BIT

curriculum affected change in these metrics among participants. Results are presented in Table 16

(appendices).

In short, we do not find evidence of changes in norms regarding these unrelated civic topics, nor do we

find evidence of changed confidence to act on environmental issues among those in the tolerance BIT

programming. We interpret this as further evidence that participants critically engaged with questions.

LIMITATIONS

We find consistent evidence that the tolerance BIT programming was efficacious in its primary goal:

training participants how to safely act in defense of social norms of tolerance and anti-violence in a

context where institutional constraints limit the menu of potential programming options. We believe

that this goal has implications for P/CVE, given that the behaviors and attitudes that are being targeted

by such a “social climate” strategy are antecedents to radicalization. Furthermore, implementing the

programming as an RCT with a placebo enabled us to conclude that changes in program participants

were attributable to the programming, while also enabling us to measure the extent to which those

changes can be attributed to program content versus the social, effects of participating in a multi-week

mixed-group experience focused on civic issues. Despite these aforementioned findings, several study

limitations should guide future research on this important topic.

First, although the programming clearly resulted in changes within participants for the outcomes we

measured, measurement activities generally took place shortly after the end of programming.

129

Thus

the long-term efficacy of programming is not currently known. Generally, there is scant evidence

examining the long-term impact of bystander intervention training programs focused on similar anti-

social behaviors.

130

As discussed, most BIT programs are designed as short-duration training sessions

(often in the form of a single workshop, or an online training). This program was deliberately designed

as a large dose (28 contact hours) with the goal of long term impact.

Second, we have taken great efforts to design and implement our measurement strategy in such a way

that mitigates the possibility that respondent answers were driven by social desirability bias and/or

demand effects. Regarding the former (which we have discussed at varying points throughout this

report), we have good reason to believe that participants felt comfortable and secure answering all

128

Banyard, V. L., et al. "How do we know if it works? Defining measurable outcomes in bystander-focused violence

prevention." Psychology of Violence 4.1 (2014): 101-15.

129

Most participants finished endline activities within about two weeks following programming end.

130

Nelson, Jacqueline K., Kevin M. Dunn, and Yin Paradies. "Bystander anti‐racism: A review of the literature." Analyses of Social

Issues and Public Policy 11.1 (2011): 263-284

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 52

questions privately and with knowledge that their responses would not be observed by any person

within their country. Importantly, the measurement activities were carried out separately from program

facilitators and Rupantar, and respondents had the opportunity to save their progress and return for a

comfortable amount of time. Moreover, our findings with respect to perceptions of the legitimacy of

VEO goals (in spite of violence) suggest to us that participants critically engaged with the questions, and

did not feel social pressures to parrot “correct answers;” indeed, many of the participants appeared

willing to agree with the premise that VEO goals can potentially be legitimate.

Regarding the latter, we deliberately designed the surveys and web-based stimulus in attempts to avoid

demand effects- the surveys, for example, were presented to respondents as research on different

personality traits and styles of leadership. The surveys themselves contained many batteries of questions

measuring personality, social characteristics, worldviews, and other traits which we deemed relevant to

the study, but which did not focus on the topics of out-group tolerance or bystander intervention

training. Moreover, participants were assured that their receipt of the completion certificate was not

tied to any of their responses to surveys. Finally, we have attempted to include behavioral measures in

the form of web-based activities that simulate (1) participant exposure and reaction to a news story

about religious persecution and (2) a follow-up activity designed to directly test intervention behaviors

in the form of a ball toss game. Unfortunately, our implementation of this second behavioral activity

yielded only half of our intended sample size, because of covid-19 related shutdowns that rendered

many participant devices incompatible with the software. We presented findings from the first activity

(despite this problem) because we observed an effect of programming, despite the underpowered

sample.

Third and finally, ultimately whether these participants actually intervene in real-world situations where

they witness intolerant and/or violent behaviors cannot easily be assessed. During scoping missions, we

asked potential participant universities whether, for example, discipline reports were available which

would include incidents of intolerant behaviors. Although the universities do discipline students for a

variety of offenses, our general sense was that they did not collect information on the outcomes that

this research focuses on.

131

In addition to this measurement issue (i.e., records of intolerant behavior),

programming would need to be scaled beyond these 6 implementation sites in order to assess broader

societal impacts.

SUMMARY

In summary, we find consistent evidence that the tolerance BIT programming improved participant’s

willingness to intervene (particularly with the use of BIT strategies when presented with a variety of

perpetrator/victim scenarios). Moreover, we find that this effect was attributable to the BIT program’s

improving of participant competence/confidence to act. While the tolerance BIT programming also

appeared to improve participant norms and skills which we believe are prerequisite to willingness to

intervene, the value-added of the tolerance BIT programming per se was not substantially greater than

that of the placebo programming, which suggests that these prerequisite norms and skills can potentially

be developed by any program that provides an inclusive social setting in which participants focus on civic

engagement. We discuss implications of this finding further in the concluding remarks of the report.

131

Moreover, soliciting the provision of private student records as a part of the research design would have been complicated

and ethnically perilous, with the potential to deter interest in programming.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 53

Regarding the type of facilitator used for the tolerance BIT programming, we do not find substantial

differences between peer leaders and traditional authority figures (namely, professors). To the extent

that we do find differences within scattered measures, the results suggest that traditional authority

figures may be better at leading programming of this nature, and that this advantage could rooted in

participant perceptions of experience and approachability, as well as (potentially) participant perceptions

that they learned better with the traditional authorities leading the classes.

8. DISCUSSION AND CONCLUSIONS

This report documents the design and evaluation of a secondary preventing/countering violent

extremism (P/CVE) strategy implemented in Bangladeshi universities from September 2019 through May

2020. The strategy tested was a tolerance-focused bystander intervention training (BIT) program, which

we chose due to its extensive prior use as a method of combating other hateful and antisocial behaviors

among young people, and due to its potential viability in the challenging political and social context of

Bangladesh. To our knowledge, this is the first program of its kind to be implemented in a non-Western

context as a randomized controlled trial. The preponderance of findings presented above suggest

programming of this nature can be impactful in increasing the likelihood participants will implement

bystander intervention strategies to promote a climate of tolerance and to challenge hateful speech and

actions that are the basis of violent extremism. The impact is due to the program’s efficacy in improving

norms, skills, perceptions of confidence, and competence to act effectively, where the curriculum itself is

especially important for confidence and competence to act.

Interestingly, many of the intermediary outcomes we deemed prerequisite to willingness to intervene

were impacted positively by both the tolerance BIT programming, and the placebo, when compared to

participants enrolled in a no-program control group. Although some caveats apply to these findings,

given that the no-program control was not subject to the same randomization protocols as the three

treatment statuses of the field experiment, we nevertheless believe this contrast is informative. The

social interactions and general prompts to think critically, and the civic focus, may have resulted in

“virtuous spillovers,” leading subjects to defend norms of tolerance and against violent extremism even

when they were not explicitly instructed on these issues.

The implications of our results are that indirect, “social climate” strategies are a compelling approach to

address violent extremism, particularly in contexts of high political sensitivity and where individual rights

are vulnerable. As we noted in the introduction, countering violent extremism is fraught by the fact such

extremism is, by its nature, rare. This “needle in the haystack” problem means any attempt to target at-

risk individuals directly may be inefficient, given the extreme difficulty of identifying them. Moreover,

strategies using group-based surveillance or otherwise target entire groups in an aggressive and coarse

manner may do more to alienate members of that group, undermining the potential for enlisting the

support of group members who oppose violence extremism. Given these challenges and our findings,

this report makes the following recommendations for future programming:

• For contexts where institutional resources, political and/or security constraints limit available

secondary P/CVE options, targeting “peer gatekeepers” with capacity-building programming offers a

promising avenue for promoting social climates hostile to violent extremism and its antecedents. In

such contexts, those that internalize (or have already internalized) a desired social norm offers a

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 54

potentially valuable resource to P/CVE practitioners if they can be trained to safely and effectively

promote and protect those norms within their social climates.

• Within sensitive contexts, programming meant to address a negative social/behavioral norm can

potentially be achieved by targeting closely related social/behavioral norms. For secondary P/CVE

programming, this implies programming does not necessarily need to explicitly target violent

extremism or specific violent extremist organizations in program content to have the desired effects

of reducing behaviors and attitudes thought to be antecedent to the problem.

• P/CVE programming targeting “peer gatekeepers” should be designed in such a way that includes

domain knowledge development as a precursor to capacity-building, with requisite time allocated for

each component.

• Scaled P/CVE programming targeting “peer gatekeepers” can likely be delivered effectively by fellow

trained peer gatekeepers without substantial loss of quality, though with the obvious caveat that

experience likely improves quality and outcomes.

What we have demonstrated is that it is possible to empower youth and create a social “climate”

against extremism through positive reinforcement of tolerance and anti-violence norms. Such a climate

should make it less likely at-risk youth travel in an unimpeded manner down the road of violent

extremism. Our results are indicative of the promise of such a strategy.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 55

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 56

APPENDIX A.

Table 4: Effects of Programming and Tolerance BIT Program on Norms of

Tolerance

132

Outcome Expectation

Gross Effect of Programming

(N=1,025)

Effect of Tolerance BIT

Curriculum (N= 1,424)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

(4)

Coeff [SE]

(5)

p-value

(6)

q-value

Intolerance (ME) - -0.12 [0.04] 0.01** -0.07 [0.03] 0.02**

Relig. Outgroup Disgust - -0.06 [0.07] 0.43 1.00 -0.11 [0.06] 0.04** 0.22

Hostile Sexism - -0.12 [0.08] 0.12 0.75 -0.1 [0.05] 0.07* 0.41

Benevolent Sexism - -0.18 [0.08] 0.02** 0.12 -0.03 [0.06] 0.57 1.00

Anti-LGBT Disgust - -0.18 [0.08] 0.02** 0.09 -0.05 [0.05] 0.27 1.00

Anti-Western Attitudes - -0.17 [0.08] 0.03** 0.21 -0.11 [0.05] 0.04** 0.22

Distrust of Foreigners - -0.25 [0.08] <0.01*** 0.01 -0.06 [0.06] 0.26 1.00

Table 5: Effects of Programming and Tolerance BIT Program on Religious

Tolerance (Web-Based Stimulus Activity)

Outcome Expectation

Gross Effect of Programming

(N=537)

Effect of Tolerance BIT

Curriculum (N= 766)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

(4)

Coeff [SE]

(5)

p-value

(6)

q-value

Mean Effects - -0.12 [0.08] 0.12 -0.11 [0.05] 0.04**

Religious Nationalism - -0.22 [0.12] 0.06* 0.26 -0.04 [0.08] 0.59 1.00

Collective Narcissism - -0.14 [0.12] 0.22 0.89 -0.14 [0.08] 0.08* 0.33

Segregation Preference - -0.12 [0.12] 0.32 1.00 -0.13 [0.08] 0.08* 0.33

Group Victimhood - -0.04 [0.13] 0.77 1.00 -0.12 [0.08] 0.11 0.46

132

***, **, and * indicate significance at the 1, 5, and 10 percent critical level for all tables.

Table 6: Effects of Programming and Tolerance BIT Program on Rejection of

Violence

Gross Effect of Programming

(N=1044)

Effect of Tolerance BIT

Curriculum

(N= 1450)

Outcome Expectation

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

(4)

Coeff [SE]

(5)

p-value

(6)

q-value

Mean Effects + -0.3 [0.06] <0.01*** -0.09 [0.04] 0.03**

Friend Disapprove + -0.39 [0.09] <0.01*** 0.00 -0.1 [0.06] 0.09* 0.44

Friend Worry + -0.31 [0.09] <0.01*** 0.00 -0.07 [0.06] 0.23 1.00

VEO Rejection + -0.47 [0.09] <0.01*** 0.00 -0.15 [0.06] 0.01** 0.06

VEO Awareness + -0.18 [0.09] 0.05* 0.25 -0.1 [0.06] 0.08* 0.42

Open to VEO

Demobilization

+ -0.25 [0.09] 0.01** 0.03 -0.04 [0.06] 0.50 1.00

Table 7: Effects of Programming and Tolerance BIT Program on

Internalized Skills

Gross Effect of Programming

(N=1044)

Effect of Tolerance BIT Curriculum

(N= 1450)

Outcome Expectation

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

(4)

Coeff [SE]

(5)

p-value

(6)

q-value

Mean Effects - 0.36 [0.07] <0.01*** 0.22 [0.05] <0.01***

No Noticing - 0.5 [0.08] <0.01*** 0.00 0.28 [0.06] <0.01*** 0.00

No ID Emergency - 0.45 [0.1] <0.01*** 0.00 0.22 [0.07] <0.01*** 0.01

No Responsibility - 0.26 [0.07] <0.01*** 0.00 0.2 [0.05] <0.01*** 0.00

Skill Deficit - 0.21 [0.09] 0.02** 0.08 0.14 [0.06] 0.02** 0.09

Audience Inhibition - 0.36 [0.07] <0.01*** 0.22 [0.05] <0.01***

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 57

Table 8: Effects of Programming and Tolerance BIT Program on Confidence

to Act

Outcome Expectation

Gross Effect of Programming

(N=1,002)

Effect of Tolerance BIT

Curriculum (N= 1,387)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

(4)

Coeff [SE]

(5)

p-value

(6)

q-value

Mean Effects + 0.39 [0.07] <0.01*** 0.2 [0.05] <0.01***

Act Religious Intolerance + 0.37 [0.07] <0.01*** 0.00 0.22 [0.05] <0.01*** 0.00

Act vs Violence + 0.47 [0.09] <0.01*** 0.00 0.2 [0.06] <0.01*** 0.01

Act vs Extremism + 0.29 [0.06] <0.01*** 0.00 0.16 [0.04] <0.01*** 0.00

Friends Act + 0.22 [0.09] 0.01** 0.05 0.09 [0.06] 0.14 0.57

Table 9: Effects of Programming and Tolerance BIT Program on Willingness

to Intervene

Outcome Expectation

Gross Effect of Programming

(N=1044)

Effect of Tolerance BIT

Curriculum (N= 1450)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

(4)

Coeff [SE]

(5)

p-value

(6)

q-value

Mean Effects + 0.22 [0.05] <0.01*** 0.12 [0.03] <0.01***

BIT in Religious Discrim. + 0.52 [0.09] <0.01*** 0.00 0.26 [0.06] <0.01*** 0.00

BIT in Eve Teasing + 0.6 [0.09] <0.01*** 0.00 0.23 [0.06] <0.01*** 0.00

BIT in LGBT + 0.51 [0.09] <0.01*** 0.00 0.18 [0.06] <0.01*** 0.02

BIT in Radicalization + 0.37 [0.09] <0.01*** 0.00 0.17 [0.06] <0.01*** 0.04

No Passivity Religious Discrim. + 0.05 [0.09] 0.55 1.00 0.08 [0.06] 0.19 1.00

No Passivity Eve Teasing + 0.06 [0.09] 0.49 1.00 0.03 [0.06] 0.66 1.00

No Passivity LGBT + 0.21 [0.09] 0.01** 0.16 0.07 [0.06] 0.24 1.00

No Passivity VEO + 0.16 [0.09] 0.07* 0.86 0.03 [0.06] 0.61 1.00

No Perpetrating Religious Discrim. + 0.07 [0.08] 0.42 1.00 0.13 [0.06] 0.04** 0.45

No Perpetrating Eve Teasing + 0.03 [0.08] 0.72 1.00 0.08 [0.06] 0.18 1.00

No Perpetrating LGBT + 0.21 [0.08] 0.01** 0.15 0.16 [0.06] 0.01** 0.11

No Perpetrating VEO

No Perp VEO

+ 0 [0.09] 0.96 1.00 0.02 [0.06] 0.78 1.00

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 58

Table 10: Participant Identification with Peer Leaders, compared to Traditional

Authority Figures, among Those Enrolled in Tolerance BIT Program

Outcome

Peer Leaders vs Traditional Authority Figures (N=778)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

Relatability 0.06 [0.06] 0.30 1.00

Experience -0.27 [0.06] <0.01*** 0.00

Youthfulness 0.13 [0.06] 0.04** 0.17

Approachability -0.12 [0.05] 0.01** 0.06

Table 11: Effect of Peer Leaders, compared to Traditional Authority Figures, on

Participant Evaluation of Social Environment among Those Enrolled in Tolerance

BIT Program

Outcome

Peer Leaders vs Traditional Authority Figures (N=778)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

Mean Effects -0.07 [0.06] 0.30

Made New Friends -0.03 [0.07] 0.67 1.00

Recommends to Friends -0.09 [0.07] 0.21 0.43

Table 12: Effect of Peer Leaders, compared to Traditional Authority Figures, on

Participant Identification with Program Goals among Those Enrolled in Tolerance

BIT Program

Outcome

Peer Leaders vs Traditional Authority Figures (N=770)

(1)

Coeff [SE]

(2)

p-value

(3)

q-value

Mean Effects -0.04 [0.06] 0.49

Learned About Tolerance -0.03 [0.05] 0.51 1.00

Learned About BIT -0.1 [0.07] 0.13 0.27

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 59

Table 13: Impact of Peer Leaders, compared to Traditional Authority Figures, on

Outcomes of Interest among Those Enrolled in Tolerance BIT Program

Outcome (Mean Effects)

Peer Leaders vs Traditional Authority Figures (N=778)

(1)

Coeff [SE]

(2)

p-value

(3)

Intolerance 0.01 [0.04] 0.92 778

Religious Supremacy -0.06 [0.07] 0.40 416

Rejection of Violence -0.05 [0.03] 0.05* 778

Skill Impediments 0.06 [0.05] 0.27 778

Confidence/Competence -0.04 [0.05] 0.40 778

Effective BIT Use -0.03 [0.04] 0.38 778

Table 14: Summary Statistics for Outcomes of Interest

133

Outcome Count Mean St. Dev Min Max

Religious Outgroup Disgust 1,451 20.22 6.46 6 42

Hostile Sexism 1,451 15.73 4.84 4 28

Benevolent Sexism 1,451 20.59 4.41 4 28

Anti-LGBT Disgust 1,451 4.18 1.92 1 7

Anti-Western Attitudes 1,451 7.31 2.76 2 14

Distrust of Foreigners 1,451 8.37 2.25 2 14

Violence: Disapprove Of Friend Involvement 1,451 10.61 2.70 2 14

Violence: Worry About Friend Involvement 1,451 8.92 3.26 2 14

VEO Rejection 1,451 22.63 5.82 5 35

Religious Nationalism 768 3.09 1.23 1 5

Religious Collective Narcissism 767 2.67 1.12 1 5

Religious Segregation Preference 768 3.67 1.76 2 10

Religious Perceived Group Victimhood 768 3.84 0.98 1 5

Failure to Notice 1,451 7.86 2.66 2 14

Failure Via Audience Inhibition 1,451 7.45 2.63 2 14

133

Unstandardized values depicted for transparency of scaling; standardized values of additive indices were used for the analysis.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 60

Outcome Count Mean St. Dev Min Max

Failure to Take Responsibility 1,451 16.74 5.64 5 35

Failure to ID Emergency 1,451 12.21 3.19 3 21

Failure Via Skill Deficit 1,451 8.31 2.74 2 14

Would Act in Religious Intolerance 1,451 34.98 7.83 9 55

Would Act vs Violence 1,451 39.76 8.17 9 59

Would Act vs Extremism 1,451 37.63 8.17 9 58

Friends would Act 1,451 58.02 11.97 11 77

BIT responses chosen Religious Discrim. Scenario 1,451 29.66 6.14 6 42

BIT responses chosen Eve Teasing scenario 1,451 5.66 1.65 1 7

BIT responses chosen LGBT scenario 1,451 5.79 1.59 1 7

BIT responses chosen VEO scenario 1,451 4.42 1.86 1 7

Would not do nothing Eve Teasing scenario 1,451 29.18 6.73 6 42

Would not do nothing LGBT scenario 1,451 29.44 6.59 6 42

Would not do nothing Religious Discrim. Scenario 1,451 27.77 5.49 6 42

Would not do nothing VEO scenario 1,451 9.49 2.68 2 14

Would not join perp Eve Teasing scenario 1,451 9.20 2.67 2 14

Would not join perp LGBT scenario 1,451 9.57 2.71 2 14

Would not join perp Religious Discrim. Scenario 1,451 9.47 2.75 2 14

Would not join perp VEO scenario 1,451 5.86 1.59 1 7

Facilitator Relatability 1,247 19.84 3.99 4 28

Facilitator Experience 1,247 33.59 5.44 7 42

Facilitator Youthfulness 1,247 11.15 2.14 2 14

Facilitator Approachability 1,451 21.75 5.57 4 28

Made New Friends 1,247 12.06 1.84 2 14

Recommends Program to Friends 1,216 6.13 0.94 1 7

Learned About Tolerance 1,218 5.86 1.09 1 7

Learned About BIT 1,451 12.12 1.84 2 14

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 61

Table 15: Standardized Baseline Covariate Balance by Attrition

Variable

(Standardized)

No-shows (192)

Mean [SE]

Drop-Outs (316)

Mean [SE]

Finishers

(1154)

Mean [SE]

Diff (t-test)

No-Shows vs

Finishers

Diff (t-test)

Drop-outs vs

Finishers

Age -0.01 [0.07] -0.17 [0.05] -0.01 [0.03] 0.00 -0.17

Male 0.59 [0.04] 0.64 [0.03] 0.58 [0.02] 0.01 0.05

Sunni 0.78 [0.04] 0.78 [0.03] 0.78 [0.01] 0.00 0.00

Econ. Welfare 0.01 [0.07] 0.09 [0.06] 0.02 [0.03] -0.02 0.07

Anger -0.09 [0.07] 0 [0.05] -0.01 [0.03] -0.08 0.01

Grit 0.07 [0.07] -0.02 [0.06] 0.04 [0.03] 0.02 -0.07

Deliberativeness 0.02 [0.08] -0.02 [0.06] 0.03 [0.03] -0.01 -0.05

Mental Health 0.15 [0.07] -0.03 [0.05] 0.03 [0.03] 0.12 -0.06

Pro-sociality 0.01 [0.08] -0.1 [0.06] -0.01 [0.03] 0.02 -0.09

Social Media Use 0.13 [0.08] 0.16 [0.06] -0.06 [0.03] 0.19** 0.22***

Family Relations 0.02 [0.08] -0.07 [0.06] 0.04 [0.03] -0.02 -0.11

Family Overseas 3.92 [0.19] 3.51 [0.13] 3.45 [0.07] 0.47 0.06

Religiosity (Reversed) 0.06 [0.07] -0.01 [0.06] -0.02 [0.03] 0.07 0.01

Diversity of Friends 0.08 [0.08] 0.02 [0.06] -0.01 [0.03] 0.09 0.03

Negative Out-Group

Interactions

-4.25 [0.18] -4.05 [0.14] -4.16 [0.07] -0.09 0.11

Critical Thinking

Exposure

0.11 [0.08] 0.02 [0.06] -0.02 [0.03] 0.13 0.04

Prior Programming

Exposure

-0.04 [0.15] -0.05 [0.06] 0.02 [0.03] -0.06 -0.07

Table 16: Gross Effect of Programming and Effect of Tolerance BIT Curriculum

on Placebo Norms and Confidence Outcomes.

Placebo Outcome

Gross Effect of Programming

(N=1025)

Effect of Tolerance BIT

Curriculum (N= 1,424)

(1)

Coeff [SE]

(2)

p-value

(3)

Coeff [SE]

(4)

p-value

Norms: Littering, Smoking and Junk Food (Reversed) -0.10 [0.31] 0.75 0.27 [0.23] 0.24

Confidence to Act: Environmental Action 0.50 [4.59] 0.91 0.33 [0.47] 0.48

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV

OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities

| 62

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013 / DRG-LER I TASKING N058

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 63

APPENDIX B.

DISCLAIMER This publication was produced for review by the United States Agency for International Development. It was

prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the

views of the United States Agency for International Development or the United States Government.

IMPACT EVALUATION OF THE OBIRODH

– ROAD TO TOLERANCE YOUTH

LEADERSHIP TRAINING PROGRAM IN

BANGLADESHI UNIVERSITIES

LONG

TERM FOLLOW-UP

AUGUST 2022

Prepared under Contract No. GS-10F-0033M / Order No. AID-OAA-M-13-00013, Tasking N058

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 65

DRG LEARNING, EVALUATION, AND

RESEARCH ACTIVITY

IMPACT

EVALUATION OF THE OBIRODH –

ROAD TO TOLERANCE YOUTH LEADERSHIP

TRAINING PROGRAM IN BANGLADESHI

UNIVERSITIES

LONG

TERM FOLLOW-UP

AUGUST 2022

Prepared under Contract No. GS-10F-0033M / Order No. AID-0AA-M-13-00013, Tasking N058

Submitted to:

USAID DRG Center

Submitted by:

Peter Vining, NORC Principal Investigator (New York University)

Cyrus Samii, NORC Principal Investigator (New York University)

Michael Gilligan, NORC Principal Investigator (New York University)

Contractor:

NORC at the University of Chicago

Attention: Matthew Parry, Program Manager

3450 East-West Highway, Suite 800

Bethesda, MD 20814

Telephone: (301) 634-5444

E-mail: Parry[email protected]

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the United States

Agency for International Development or the United States Government

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 66

TABLE OF CONTENTS

1. EXECUTIVE SUMMARY 68

2. LONG TERM FOLLOW-UP INTRODUCTION 68

EVALUATING LONG-TERM IMPACT THROUGH RETROSPECTIVE AND

DEMONSTRATED USE OF TRAINING 69

VIOLENCE REJECTION NORMS – ADDED MODULES ON THE JUSTIFIABILITY OF

VIOLENCE 70

WILLINGNESS TO INTERVENE – ADDED MODULES ON RETROSPECTIVE NOTICING

OF AND INTERVENTION IN REAL LIFE AND SOCIAL MEDIA 71

3. LONG TERM FOLLOW-UP METHODOLOGY 73

SAMPLE SIZE, DATA COLLECTION STRATEGY, TIMELINE, AND ANALYSIS 73

BASELINE CHARACTERISTICS COVARIATE BALANCE BY LTF ATTRITION 74

DATA ANALYSIS 76

4. LONG TERM FOLLOW-UP RESULTS 77

NORMS 77

NORMS OF TOLERANCE AND INTOLERANCE: LONG TERM EFFECTS OF

PROGRAMMING 77

NORMS REGARDING THE USE AND JUSTIFIABILITY OF VIOLENCE: LONG TERM

EFFECTS OF PROGRAMMING 79

SKILLS: LONG TERM EFFECTS OF PROGRAMMING 81

CONFIDENCE AND SELF-ASSESSED COMPETENCE TO ACT 82

WILLINGNESS TO INTERVENE 83

SUMMARY 85

5. LONG TERM FOLLOW-UP DISCUSSION AND CONCLUSIONS 85

APPENDICES 87

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 67

TABLES

Table 1: Long Term Follow-up Data Collection Timeline 74

Table 2: Standardized Baseline Covariate Balance by Attrition: Long Term Follow-up vs Endline 75

Table 3: Long Term Effects of Programming and Tolerance BIT Program on Norms of Tolerance 87

Table 4: Long Term Effects of Programming and Tolerance BIT Program on Rejection of

Violence 88

Table 5: Long Term Effects of Programming and Tolerance BIT Program on Justifiability of

Violence Beliefs (All Participants) 88

Table 6: Long Term Effects of Programming and Tolerance BIT Program on Dominant In-Group

Justifiability of Violence Beliefs (Jihadist Violence Included; Muslim Participants Only) 89

Table 7: Long Term Effects of Programming and Tolerance BIT Program on Internalized Skills 89

Table 8: Long Term Effects of Programming and Tolerance BIT Program on Confidence to Act 90

Table 9: Long Term Effects of Programming and Tolerance BIT Program on Willingness to

Intervene 91

Table 10: Long Term Effects of Programming and Tolerance BIT Program on Retrospective

Evaluations of BIT Use in Real Life and on Social Media 92

Table 11: Summary Statistics for Long Term Follow-Up Additional Outcomes of Interest 93

FIGURES

Figure 1: Long Term Effects of Programming and Tolerance BIT Curriculum on Norms of

Tolerance 78

Figure 2: Long Term Effects of Programming and Tolerance BIT Curriculum on Rejection of

Violence and VEOs 79

Figure 3: Long Term Effects of Programming and Tolerance BIT Curriculum on Beliefs Regarding

the Justifiability of Violence (All Respondents) 80

Figure 4: Long Term Effects of Programming and Tolerance BIT Curriculum on Beliefs Regarding

the Justifiability of Violence (Justifiability of Jihadist Violence Module Included; Muslim

Respondents Only) 81

Figure 5: Long Term Effects of Programming and Tolerance BIT Curriculum on Internalized BIT

Skills 82

Figure 6: Long Term Effects of Programming and Tolerance BIT Curriculum on Confidence Act 83

Figure 7: Long Term Effects of Programming and Tolerance BIT Curriculum on Willingness to

Intervene 84

Figure 8: Long Term Effects of Programming and Tolerance BIT Curriculum on Retrospective

Bystander Intervention Behaviors 85

Figure 9: Pathways from BIT Programming to Willingness to Intervene 87

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 68

1. EXECUTIVE SUMMARY

This addendum details findings from long-term follow-up research with participants in the USAID-

sponsored Obirodh: Road to Tolerance bystander intervention training (BIT) randomized controlled trials,

held in six Bangladeshi Universities during late 2019 and early 2020. The endline evaluation found that

the programming significantly improved participants’ norms of tolerance and their expected willingness

and capacity to intervene when witnessing intolerant and extremist behaviors. Following these positive

impacts on program participants, the USAID DRG Center commissioned this long-term follow-up

research to understand which (if any) of these effects persisted roughly 20 months after the final wave of

programming concluded. The key findings from this long-term follow-up research are as follows:

• BIT programming had enduring effects on participants’ expected willingness to intervene when

witnessing intolerant and extremist behaviors.

• Improvements in norms of tolerance also sustained among program participants (when

compared to those in a no-program control group). However, these sustained improvements

were also present among participants in a placebo program (that focused on transportation,

environmental, and public health challenges).

In short, while the tolerance BIT program was successful in having long-term efficacy in training

participants to have confidence and thus willingness to intervene, it appears that long-term

improvements in tolerance norms can also be sustained indirectly through programming that facilitates

improved critical thinking skills and dialogue among mixed groups of young people.

2. LONG TERM FOLLOW-UP INTRODUCTION

A key finding from the original impact evaluation is that the program significantly improved participants’

norms of tolerance, and their expected confidence, willingness, and capacity to intervene safely and

effectively when they witness intolerant and extremist behavior. While encouraging, these findings were

based primarily on endline data collected approximately one to two weeks after the end of

programming (during late 2019 for wave 1 participants, and early 2020 for wave 2 participants). The

findings thus do not shed light on the longer-term impact of such programming, or whether changes

within the social micro-climates of participants have endured following, for example, the disruptions of

the Covid-19 pandemic, or the growth and maturation of the young participants themselves.

This addendum details the results of a long-term follow-up of those enrolled in the treatment

programming, as well as those enrolled in placebo and no-program control. The long-term follow-up

was conducted from January-March, 2022, and successfully surveyed 1,013 individuals who participated

in programming or no-program control. In summary, we find an enduring effect of the tolerance

bystander intervention training (BIT) curriculum on participants’ confidence to act when presented with

scenarios of religious intolerance, scenarios of violence, and scenarios of extremism. Moreover, we find

some additional suggestive evidence that this increased confidence to act may be resulting in a greater

willingness to intervene. While we find numerous other enduring effects of programming on norms of

tolerance and norms against violence, they cannot be attributed specifically to the tolerance BIT

curriculum. Finally, while we investigated long-term differences between those enrolled in peer and

authoritative facilitator groups within the tolerance BIT programming, we did not find any substantively

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 69

meaningful differences. For brevity, we focus this analysis on comparing the tolerance BIT program to

placebo and no-program control.

In addition to surveying program participants who were willing to participate in the long-term follow-up

roughly two years after the programming ended, an attrition analysis was conducted to investigate

whether 2022 long-term follow-up participants systematically differ from 2019-2020 program

participants who submitted an endline on outcomes of interest, but who did not participate in the long-

term follow-up. We find minimal differences between these groups in baseline covariates. Moreover, we

did not find an association between treatment assignment and participation in the long-term follow-up.

The following sections detail the design and data collection for the long-term follow-up, as well as

results and discussion.

EVALUATING LONG-TERM IMPACT THROUGH RETROSPECTIVE AND DEMONSTRATED

USE OF TRAINING

A survey framework nearly identical to that used in endline data collection was developed for

conducting the long-term follow-up. Regarding outcomes of interest, we replicate question modules

used for the main results presented in the final report. These include the measures of tolerance and

rejection of violence norms, skills, confidence/competence to act, and willingness to intervene. In

addition to replicating these primary outcomes of interest, we developed and employed several

additional question modules designed to further probe participant norms regarding the justifiability of

violence. Moreover, we developed and employed several measures designed to elicit retrospective

estimations of whether and how many times the respondent noticed and/or acted (either in real life

and/or over social media) in response to situations characterized by religious intolerance, sexism,

extremism, and the use of violence.

Finally, we replicated the web-based stimulus activity (designed to test whether programming had an

inoculative effect against in-group collective narcissism and justifiability of violence in defense of one’s

religious in-group) using a contemporaneous international news excerpt detailing sectarian violence in

either India or Bangladesh. Variants of the activity (for Muslims, Hindus, Christians, and Buddhists) were

created using the same article structure, but adapted using contemporaneous acts of sectarian violence

targeting religious minorities within the region. (So, for example, anti-Hindu violence within Bangladesh

itself was presented to Hindu participants; in contrast, anti-Muslim violence in India was presented to

Muslim participants). In adapting the article, the researchers took care to avoid deception of any kind; all

versions detailed real and recent acts of sectarian violence within Bangladesh and/or India that featured

similar anecdotes (such as a viral video of a person being attacked). The stimulus activity was followed

by a reading comprehension test meant to serve as an attention check, and then followed by survey

modules measuring religious collective narcissism and justifiability of violence.

To accommodate these aforementioned survey additions and other changes, nearly all programmatic

and facilitator-related questions were removed from the long-term follow-up survey, given that we

observed few differences between authoritative and peer-led groups at endline. In their place, we ask

respondents (who participated in programming) a brief set of questions measuring retrospective

program salience; we did not observe differences between treatment and placebo on these metrics, and

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 70

do not present them in the analysis.

The following section details each of the new measures

incorporated into the survey framework.

VIOLENCE REJECTION NORMS – ADDED MODULES ON THE JUSTIFIABILITY OF VIOLENCE

In addition to tolerance norms, a major focus of the study was whether programming had an impact on

participant norms regarding the use of violence. At time of programming, it was decided that only

indirect questions about the acceptability/justifiability of violence were to be used in endline surveys, out

of an abundance of caution against the possibility that program participants (some of whom were sharing

devices to complete surveys) might inadvertently reveal and/or be observed providing responses to

sensitive violence-related questions. Researchers thus intentionally avoided including any named violent

extremist organizations (VEOs) within programming materials or measurement activities during program

implementation, due to the risk that attention placed on the programming by authorities (as well as the

small group dynamics of the programming) would potentially put participant safety in question should

such questions be asked of them.

For the long-term follow-up, we incorporated three additional question modules that more directly

measure participant perceptions regarding circumstances under which violence might be justified, though

we continue to avoid the use of any questions that prompt the respondent to evaluate a single specific

named VEO.

These additional violence-related questions were asked only during long-term follow-up,

after it was determined that the risk of a participant's responses being observed by others was very low,

and fully under control of the participant himself/herself, given that the survey was conducted remotely

without using enumerators. The additional modules are as follows:

• Support for Violence in Defense of Just Causes: An index of four items measuring the

participant's belief that violence is sometimes necessary or justified if in defense of a just/worthy

cause, or in defense of one's values or beliefs. These questions were derived from a 2021 study

of violent extremist attitudes in young adults by Nivette (2021).

These questions were phrased

without reference to a particular cause, faith tradition, or other identity group and thus asked of

all participants.

• Support for Jihadist Violence (Muslim Participants Only): An index of seven items asked

only of participants who identified as Muslim, measuring agreement with statements focused on

whether violence carried out by an assortment of international jihadist groups can be justified.

Specifically, we asked those in tolerance BIT programming and placebo whether (1) they had remained friends with at least

one person from their leadership training sessions, and (2) whether they remained in contact with their facilitator to construct a

retrospective programmatic salience index. We also asked program participants how many times they had used their training

over the past 18 months (without specifying which type of training). We found no difference between treatment and placebo

on these measures, suggesting that both those assigned to the tolerance BIT treatment and those assigned to the placebo

curriculum felt roughly the same about the program’s enduring salience.

One question within the “support for jihadist violence” module, asks participants to indicate level of agreement with the

statement “[t]he fighting of groups like the Taliban, Al Qaida, and ISIS is justified.” We view this question as the most sensitive

violence-related question asked within the long-term follow-up survey framework. In contrast (and for demonstrative

purposes), violence-related questions in endline were phrased in ways such as “It's okay to agree with the goals of violent

extremist groups like ISIS but disagree with their methods of pursuing those goals” and “[t]he goals of violent extremist groups

like ISIS are popular in Bangladesh.”

Nivette, Amy, et al. "Understanding changes in violent extremist attitudes during the transition to early adulthood." Journal of

Quantitative Criminology (2021): 1-30.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 71

These questions were derived from a recent study of sacred values and vulnerability to violent

extremism by Gomez et al (2017).

• Support for Violence on Behalf of Religious In-Group: An index of seven questions

evaluating the participant’s agreement with statements related to the justification and

defensibility of using force in defense of one’s religious tradition, adapted from Swann et al

(2009).

These questions were added to the web-based stimulus activity post-activity survey, and

adapted such that the in-group being referenced matched the religious tradition indicated earlier

in the survey by the participant. All participants were thus presented with this question module.

For the subsequent long-term follow-up, these question modules were combined with the index

measuring collective narcissism following the stimulus activity, as well as the violence worriedness index

(which measured worriedness following a friend’s hypothetical engagement with violent activities), both

of which were directly replicated from endline activities.

Altogether, these measures capture the extent to which participants reject violence as a legitimate

means of pursuing goals. It is expected that evidence of program efficacy would be reflected in relatively

lower levels of these measures in those receiving tolerance-focused BIT treatment when compared to

those in placebo and no-program control groups.

WILLINGNESS TO INTERVENE – ADDED MODULES ON RETROSPECTIVE NOTICING OF AND

INTERVENTION IN REAL LIFE AND SOCIAL MEDIA

For the long-term follow-up, indicators of effective intervention on behalf of a victim were replicated

using the same series of vignettes that were employed during endline data collection activities.

Moreover, with the benefit of more than 18 months during which the participant might have potentially

drawn on this training in real-world (or social media) situations where intervention may have been

helpful, we sought to directly ask respondents to provide retrospective assessments of any BIT

behaviors during the 18 months prior to the long-term follow-up (LTF) survey. Retrospective BIT

behaviors were measured using counts of BIT since the end of programming. The retrospective use

measures were developed by the researchers, and worded to avoid referencing the programming, or

BIT itself (to accommodate those in the no-program control).

Participants were asked the following:

• Noticing (Any): Participants were first asked a screener question measuring if they had ever

noticed any incidents of religious intolerance, gender intolerance, extremist beliefs, and/or

violent incidents in real life or on social media during the past 18 months.

• Noticing in Real Life: Participants who responded in the affirmative to having noticed any of

the aforementioned incidents were asked to estimate the number of times they noticed them in

real life during the past 18 months.

Gómez, Ángel, et al. "The devoted actor’s will to fight and the spiritual dimension of human conflict." Nature Human Behaviour

1.9 (2017): 673-679.

Swann Jr, William B., et al. "Identity fusion: the interplay of personal and social identities in extreme group behavior." Journal of

Personality and Social Psychology 96.5 (2009): 995.

A separate and smaller module of questions was asked of all program participants to assess whether their training had been

used; in this module, type of training was not specified (in order to accommodate both BIT treatment and placebo). This

module was used to measure enduring programming salience.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 72

• Noticing on Social Media: Participants who responded in the affirmative to having noticed

the aforementioned incidents were asked to estimate the number of times they noticed them on

social media during the past 18 months.

• Acting (Any): Participants were asked a screener question measuring if they had ever acted in

response to any incidents of religious intolerance, gender intolerance, extremist beliefs, and/or

violent incidents in real life or on social media during the past 18 months.

• Acting in Real Life: Participants who responded in the affirmative to having acted in response

to the aforementioned incidents were asked to estimate the number of times they had done so

in real life, for each type of incident during the past 18 months.

• Acting on Social Media: Participants who responded in the affirmative to having acted in

response to the aforementioned incidents were asked to estimate the number of times they had

done so on social media, for each type of incident during the past 18 months.

From these aforementioned question modules, four indices were constructed measuring retrospective

bystander intervention (noticing and acting in real life and on social media) for scenarios characterized

by (1) religious intolerance, (2) sexism, (3) extremism, and (4) violence. These four indices are then used

in the subsequent analysis. Finally, while we were unable to directly measure interventions occurring in

participants’ day-to-day lives, we added a behavioral measure into this module of questions to give

participants an opportunity to show an example of a time when they noticed or acted in response to

these types of scenarios over social media, in the form of a screenshot from their personal social

media.

• Willingness to Upload a Screenshot of Intervention on Social Media Finally, we asked

respondents who indicated having intervened over social media in the past 18 months if they

would be able and willing to share a screenshot of having done so. Pending a response of “yes,”

the respondent was then able to upload a photo through Qualtrics’ file upload modality. The

yes/no response to this question, in addition to whether the respondent actually uploaded a file

(but not the content of the file), were both used to construct an index measuring this behavioral

outcome.

We did not define the specific social media venue to give the participants room for interpretation; however, the screenshots

we received were overwhelmingly from Facebook.

A substantial proportion of respondents who indicated willingness to upload a screenshot did not actually upload one (before

continuing to finish the survey). The index thus ranges from 0-2, with a “2” indicating that a screenshot was actually uploaded,

and a “1” indicating willingness without subsequent follow-up. Note that to respect participant privacy, we do not evaluate or

share the specific contents of screenshots within this analysis.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 73

3. LONG TERM FOLLOW-UP METHODOLOGY

SAMPLE SIZE, DATA COLLECTION STRATEGY, TIMELINE, AND ANALYSIS

For evaluating program efficacy in the long-term follow-up, we maintained the same assumptions and

desired sample sizes in order to compare efficacy of programming to placebo, and attempted to recruit as

many former participants (including drop-outs) who comprised the population of those who submitted

baseline survey data for programming. This amounted to a sample frame of approximately 2,212 individuals

who were randomized into one of the four treatment statuses (i.e., the two facilitator variants of tolerance

BIT programming, placebo, or no-program control). Of those 2,212 participants, approximately 1,305

were program participants who were awarded a certificate of completion by Rupantar (the Bangladeshi

civil society organization implementing the training) for having completed programming (treatment or

placebo), 560 were participants who were randomized into a treatment group and submitted baseline data,

but either never started programming or dropped out during programming, and 347 were no-program

control participants (204 of whom submitted an endline survey).

Data collection for the long-term follow-up was carried out from December 2021 through March 2022.

The 2,212 individuals were first sent a brief opt-in intake form collecting updated contact information and

interest in participating in the long-term follow-up survey. Existing email addresses and phone numbers

collected during 2019-2021 program implementation were used to initiate contact for this purpose, with

local assistance provided by the seven technical coordinators hired by NORC at the University of Chicago

to facilitate local data collection efforts within/around each treatment site. A small incentive in the form of

a mobile phone recharge (BDT 200, worth approximately $2.33) was offered for completing these data

collection activities. This approach yielded intake form submissions from 1,229 of the sampling frame.

Intake recruitment efforts spanned approximately two weeks in late December, 2021.

Those who opted in via the intake form were later sent the survey starting in early January, 2022.

Updated contact information was used to send periodic reminders via email, SMS, and phone calls, with

a limit of three contact attempts per participant. The deadline for these data collection activities was

extended through March, due to new Covid-related university closures in Bangladesh.

The average

completion time for long-term follow-up activities was approximately one hour, which was roughly

about the same amount of time required to complete the endline survey. This amount of time was not

thought to impact data quality, evidenced by low levels of "survey speeding,”

and low rates of failure on

four attention check/distraction questions.

An additional 34 unique individuals submitted intake forms who were not associated with the programming/previous data

collection efforts; presumably some participants shared the recruitment link with friends. Recruitment verbiage explained that

anyone who submitted at least one prior survey during 2019-2020 was eligible to participate.

At the time, the Omicron variant of Covid-19 had begun circulating; while most participants were able to complete data

collection activities remotely without issues, some had been relying on university WIFI connections.

Completion times of less than 25 percent of the completion median (14 minutes versus 56 minutes) among those estimated

to have completed the survey in fewer than three hours (a conservative proxy for completion in a single setting) were

disqualified from analysis. From this measure, only 8 participants were classified as “speeders,” which was less than one

percent of respondents.

Respondents were asked to complete four simple arithmetic problems that served as both attention checks and distraction

tasks while transitioning into new sections of the survey. Inclusion criteria for the analysis required passing at least three of four

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 74

All data were collected through Qualtrics and were stored on secure US-based servers. Participant

personal identifiable information (PII, in the form of phone numbers) were used to match participants to

de-identified participant numbers that were generated and assigned by the researchers during data

processing of baseline/endline data during 2019-2020 programming. Data cleaning and translation took

place during April, 2022.

Table 1 summarizes the timing of long-term follow-up data collection for each treatment status and

wave of the study. Data collection totals for combined endline activities are depicted for reference.

Table 1: Long Term Follow-up Data Collection Timeline

Treatment Placebo

Peer

Tolerance

Prof/

Tolerance

Placebo

Peer/

Tolerance

Prof/

Tolerance

Pure

Control

Wave 1 1 1 2 2 2 2

Endline

(Combined)

December

2019/ January

2020

(N = 185)

December

2019 /January

2020

(N= 190)

December

2019/ January

2020

(N= 190)

March 2020

(N=221)

March 2020

(N=227)

March 2020

(N=229)

March 2020

(N=204)

Long Term

Follow-up

January-March

2022

(N=130)

January-March

2022

(N=136)

January-March

2022

(N=139)

January-March

2022

(N=171)

January-March

2022

(N=168)

January-March

2022

(N=181)

January-March

2022

(N=139)

BASELINE CHARACTERISTICS COVARIATE BALANCE BY LTF ATTRITION

Table 2 displays baseline covariate balance between those who submitted a long-term follow-up survey

(complete or partial responses, N=1,064), and those who submitted an endline survey in 2019/2020, but

did not complete any stage of the long-term follow-up survey (N=574).

Imbalances between these

groups were identified with respect to gender (LTF respondents were more likely to be male) and age

(LTF respondents were more likely to be younger) than endline submitters who did not complete the

LTF. In addition, minor differences were found with respect to baseline measures of economic welfare

(LTF respondents were less economically well-off), and grit (LTF respondents had higher baseline levels

of grit). Finally, LTF attrition status was not found to be correlated to treatment assignment; LTF

respondents were well-balanced on treatment assignment when compared to endline submitters who

of these attention check questions; only two of 1,064 complete or partial long-term follow-up participants failed these

attention checks, which was less than one percent of respondents.

Approximately 188 respondents completed a baseline survey and a long-term follow-up survey, but did NOT complete an

endline survey. Only 42 of these respondents had been awarded completion certificates and are most likely to have failed to

complete endline responses due to 2020 covid-related shutdowns. The bulk of these respondents (107) were drop-outs and

never-starters who were assigned to treatment, but did not complete it. These respondents were presumably motivated by the

small incentive offered for completing the long-term follow-up. In order to explore whether the addition of these long-term

follow-up participants who did not complete endline activities may have impacted results, all model specifications were

replicated for (1) only those endline participants who received a certificate (and no-program control), (2) re-classifying never-

starters assigned to treatment as no-program control, and (3) limiting the analysis only to those participants that also

submitted an endline. Results were generally robust across these different scenarios (with minor changes in coefficients), albeit

loss of statistical significance on some terms presumably due to smaller sample sizes.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 75

did not complete the LTF. Unbalanced demographics were included in the subsequent analysis as

covariates.

Table 2: Standardized Baseline Covariate Balance by Attrition: Long Term Follow-up vs

Endline

Variable (Standardized)

LTF (1,064)

Mean [SE]

No LTF Endline (574)

Mean [SE]

Diff (t-test)

LTF vs No LTF Endline

Age -0.05 [0.03] 0.06 [0.04] -0.12**

Male 0.63 [0.02] 0.55 [0.02] 0.08***

Sunni 0.77 [0.01] 0.79 [0.02] -0.03

Economic Welfare -0.04 [0.03] 0.1 [0.04] -0.13**

Anger 0.01 [0.03] 0 [0.04] 0.01

Grit 0.07 [0.03] -0.09 [0.04] 0.17***

Deliberativeness -0.01 [0.03] -0.02 [0.04] 0.01

Mental Health 0.04 [0.03] -0.01 [0.04] 0.05

Pro-sociality 0.02 [0.03] -0.03 [0.04] 0.05

Social Media Use -0.05 [0.03] 0 [0.04] -0.05

Family Relations 0.02 [0.03] 0 [0.04] 0.02

Family Overseas 3.37 [0.07] 3.46 [0.09] -0.09

Religiosity (Reversed) -0.04 [0.03] 0.02 [0.04] -0.06

Diversity of Friends 0 [0.03] -0.01 [0.04] 0.01

Negative Out-Group Interactions -0.13 [0.00] -0.13 [0.00] -0.00

Critical Thinking Exposure -0.02 [0.03] -0.09 [0.04] 0.07

Prior Programming Exposure -0.02 [0.04] 0.01 [0.05] -0.02

Note: Robust standard errors in brackets. The value displayed for t-tests are the differences in the means across the groups. All estimation

regressions include school and wave fixed effects. In the right-hand column, ***, **, and * indicate significance at the one, five, and ten

percent critical level, respectively.

CONTRACT No. GS-10F-0033M / ORDER No. AID-OAA-M-13-00013

USAID.GOV OBIRODH Impact Evaluation: Road to Tolerance Youth Leadership Training Program in Bangladeshi Universities | 76

DATA ANALYSIS

For the long-term follow-up, we replicate the same analysis framework articulated in the final endline

report. Specifically, regarding the effect of the program on outcomes of interest, we report estimates

from regression models specified as follows:

Endline Outcome = a + b*BIT Curriculum + c*Baseline Outcome + University FE

+ Wave FE + Unbalanced Covariates + e (1)

where a is an intercept term that measures mean outcomes in the absence of the BIT intervention, b

measures the effect of the BIT intervention, c measures how the baseline measures of the outcomes

correlate with endline measures, the University FE and Wave FE terms refer to indicator variables that

control for variation at the level of universities and data collection waves, and the Unbalanced Covariates

are control variables determined on the basis of analysis of pre-treatment covariate data, and e is an

error term that captures other idiosyncratic variation from individual to individual. We estimate the

regression using weighted least squares, weighting by the inverse probability of treatment assignment.

We use heteroskedasticity robust standard errors, given that the treatment assignment was at the level

of individuals.

We report two types of effects. The first effect is based on the contrast between tolerance BIT program

participants versus those in the no-program control condition. We do this by estimating equation (1) on

a sample that includes program participants and those who were not engaged in any program at all, and

excludes those who received the placebo curriculum. This estimates the gross impact of the program,

including the effect of the BIT curriculum per se, as well as any effects attributable to getting people

together in a novel social situation to think critically. The second effect isolates the effect of receiving

the BIT curriculum as compared to receiving the placebo curriculum. This effectively controls for the

aforementioned social interaction effects, which would be operative in both the BIT and placebo

curricula (therefore enabling comparison of differences between the tolerance BIT and placebo). To do

this, we estimate specification (1) on a sample that includes those in the BIT curriculum and those in the

placebo curriculum, excluding those who were not engaged in any program at all. This estimates the

added value of the curriculum per se, above and beyond the effect of the novel social interactions and

critical thinking exercises that were also included in the placebo curriculum.

We use a hierarchical testing and statistical significance strategy to first evaluate overall effects on sets of

items, and then to evaluate item-level effects that are adjusted to account for the fact that we are doing

many tests (i.e., to address the “multiple comparisons problem”). As in the final report, we have sets of