Treatment Implementation Following Behavioral Consultation in Schools: A Comparison of Three Follow-up Strategies

Abstract. This study examined teachers' implementation of treatment plans following consultation. Interventions were implemented for 45 elementary school students referred for consultation and intervention due to academic concerns, challenging behavior, or a combination of the two. The consultation follow-up procedures examined were brief weekly interviews, weekly interviews combined with an emphasis on the commitment to implement the treatment, and performance feedback. Performance feedback was associated with superior treatment implementation and child behavioral outcomes when compared to the two other conditions. Treatment implementation did not differ for the weekly follow-up meeting and the commitment emphasis conditions at a statistically significant level. Teacher ratings of consultants and treatment acceptability were similar across conditions. A moderate statistically significant correlation between treatment integrity and child behavioral outcome was obtained. The correlation between treatment acceptability and implementation was quite small and was not statistically significant. The implications of these findings for consultation and intervention are discussed.

The dominant psychological treatment strategies for children's social, emotional, behavioral, and academic concerns can broadly be described as environmentally based or behavioral. This reflects the accumulated research suggesting that effective treatment strategies for children include modification of variables in the child's natural environments (DuPaul & Eckert, 1997; Swanson & Hoskyn, 1998; Weiss & Weisz, 1995; Weisz, Weiss, Alicke, & Klotz, 1987). As a result, effective intervention for children often begins with changing the behavior of parents, teachers, and other care providers. Changing adult care providers' behavior is typically necessary because it is rarely practical to have a therapist directly implement treatment across weeks or months. These realities converge to create one of the central challenges confronting school psychologists treating children. How can a consulting psychologist influence adults' behavior so that an effective treatment program will be implemented?

The extent to which a treatment plan is implemented as designed has been described as treatment integrity (Peterson, Homer, & Wonderlich, 1982) or treatment fidelity (Moncher & Prinz, 1991). Although the literature examining the relationship between treatment integrity and treatment outcomes is limited, an intuitive relationship appears to exist. Interventions that are poorly implemented appear less likely to be effective. The limited evidence suggests the importance of treatment integrity in achieving successful treatment. Researchers have provided descriptive evidence suggesting the relationship between treatment integrity and child outcome across such diverse interventions as the cognitive-behavioral treatment of anxiety (Vermilyea, Barlow, & O'Brien, 1984), class-wide peer tutoring (Greenwood, Terry, Arreaga-Mayer, & Finney, 1992), social skills intervention (McEvoy, Shores, Wehby, Johnson, & Fox, 1990), and multisystemic therapy for juvenile offenders (Henggeler, Melton, Brondino, Scherer, & Hanley, 1997). Although these studies did not manipulate treatment integrity directly, they all assessed treatment integrity in some manner and demonstrated a positive relationship between the degree of treatment implementation and client outcome. Other studies that have experimentally manipulated treatment integrity have both supported the importance of accurate treatment implementation (Noell, Gresham, & Gansle, 2002) and demonstrated that implementation of all elements of a treatment plan may not be critical to success (Gansle & McMahon, 1997).

The need to assure treatment implementation naturally raises the question: What can be done to assure accurate implementation of interventions? Assuring plan implementation in consultation and behavior therapy is frequently more challenging than developing a treatment plan with a high probability of success (Foxx, 1996; Noell, Duhon, Gatti, & Connell, 2002). However, in comparison to the literature examining psychological treatments for childhood concerns, the literature examining means of ensuring sustained accurate implementation of treatments is comparatively poorly developed (Noell, Duhon et al., 2002; Sheridan & Gutkin, 2000). A technology for assuring plan implementation is the critical enabling technology that can permit the effective use of a wide range of academic and social intervention strategies.

Although it may have some unique features, enhancing treatment implementation by care providers is an extension of the broader problem of adult behavior change. Implementing the school- or home-based portion of a child's treatment plan may have many features in common with other performance management contexts such as diet and exercise programs or employee performance management. Specifically, these behaviors may be new to the individual, they are effortful, they may require resources the person lacks, and they exist in an environment in which multiple opportunities and demands compete for the adult who is being asked to implement them. Conceptualizing treatment implementation as a specific instance of the broader problem of adult behavior change suggests the utility of procedures that have been primarily researched outside of educational and child treatment contexts such as performance feedback.

Performance feedback has been extensively researched in employment and institutional settings as a means of initiating and sustaining adult behavior change (Alvero, Bucklin, & Austin, 2001; Balcazar, Hopkins, & Suarez, 1985). Performance feedback consists of monitoring a behavior that is the focus of concern and providing feedback to the individual regarding that behavior. Additional elements such as goal setting, performance contingencies, and graphic displays of performance have been found to enhance the efficacy of performance feedback (Alvero et al., 2001; Balcazar et al., 1985). For example, two related studies have demonstrated that teachers provided more praise when they were provided feedback on how often they praised students while implementing an intervention plan (Jones, Wickstrom, & Friman, 1997; Martens, Hiralall, & Bradley, 1997). Additionally, the Jones et al. study found that implementation did not improve following consultation until feedback was provided.

A series of recent studies have examined several follow-up procedures for consultation that included variations on the use of performance feedback (Mortenson & Witt, 1998; Noell, Duhon et al., 2002; Noell, Witt, Gilbertson, Ranier, & Freeland, 1997; Noell et al., 2000; Witt, Noell, LaFleur, & Mortenson, 1997). In each of these studies, behavioral treatment plans were developed for children due to academic or behavioral concerns. Teachers implemented the treatment plans following training for themselves, the student, and any other participants (e.g., peer tutors). Across studies, intervention implementation was typically initially high, but decreased over time, and in many instances the deterioration was precipitous. The introduction of performance feedback that included graphic presentation of treatment integrity and child performance was associated with substantial improvements in implementation.

This series of studies examined the importance of procedural elements of performance feedback as it has been adapted to behavioral consultation. Specifically, substantial initial training of the teachers did not appear to be necessary for performance feedback to be effective (Noell et al., 1997). Weekly performance feedback was effective, but less consistently so than daily feedback (Mortenson & Witt, 1998). Brief follow-up interviews that did not include data review were less effective than performance feedback (Noell et al., 2000). Follow-up that reviewed implementation without graphing the time course of implementation and child outcome was less consistently effective than when follow-up included graphing (Noell, Duhon et al., 2002). Additionally, implementation remained at generally high levels as the frequency of follow-up contacts was thinned.

The studies summarized above showed that implementation of intervention plans by teachers generally deteriorated to very low levels in the absence of structured follow-up (Mortenson & Witt, 1998; Noell, Duhon et al., 2002; Noell et al., 1997; Noell et al., 2000; Witt et al., 1997). The studies demonstrated the efficacy of performance feedback that includes graphic presentation of implementation and child outcome in sustaining treatment integrity. These studies differed from the bulk of the performance feedback literature because the consulting school psychologist held no administrative authority and did not clearly possess higher status within the organization than the teacher receiving the feedback. Performance feedback was effective despite the absence of a hierarchical relationship.

In contrast to performance feedback, the social influence literature suggests alternative methods for enhancing treatment implementation that emphasize antecedents and the interpersonal nature of consultation (O' Keefe & Medway, 1997). From an interpersonal perspective, treatment implementation in consultation is a problem of the correspondence between the commitment to implement treatment and subsequent behavior. Commitment consistency research has examined other socially desirable behaviors, but has not been applied to treatment implementation. Research in contexts outside the treatment of children has suggested the potential utility of emphasizing reciprocal obligations and emphasis on consistency with prior commitments (Howard, 1995). Research has also found that training communicators to speak vividly, personalize information, induce commitments, and emphasize risks of negative outcomes in recommendations was effective in influencing behavior (Gonzales, Aronson, & Costanzo, 1988; Lipsitz, Kallmeyer, Ferguson, & Abas, 1989).

The current study was designed to address two major gaps in the existing literature regarding treatment implementation following consultation in child behavior therapy within schools. First, little if any research exists directly examining the effect of overt social influence messages within consultation on treatment implementation. In particular, social influence bids that overtly include discussion of potential barriers to implementation, emphasize commitment to the child, discuss negative consequences associated with nonimplementation, and include proactive planning for implementation appear to be promising, but have not been examined. Second, the existing literature supporting the efficacy of performance feedback in consultation has relied exclusively on small n or single subject designs. This study is the first randomized field trial of these procedures. This study sought to examine the following research questions. First, to what extent would weekly follow-up, social influence, and performance feedback lead to differing levels of treatment implementation following consultation? Second, to what extent would condition assignment and level of treatment integrity be associated with student behavior change? Third, would teacher perceptions of intervention acceptability, intervention effectiveness, of consultants vary based on the type of follow-up provided? Finally, what is the relationship between treatment integrity, treatment acceptability, student behavior change, and teacher ratings of student concerns?

Participation in this study was initiated when teachers referred students to a school-based team that provided consultative psychological services and intervention planning for students who were experiencing academic or behavioral difficulties at school. The school-based teams were composed of doctoral students in school psychology who were supervised by the first two authors. Seven doctoral students who were predominantly female (5 of 7) and Caucasian (6 of 7) acted as consultants. The non-Caucasian consultant was African American. School-based teams typically existed within a school for 1 academic year and were a general resource for problem solving. All of the referrals to the teams that were potentially eligible for inclusion were included in this study; however, the teams provided a broader array of supports and services as needs and requests arose within the schools.

Consultants received extensive training in consultation, the measurement procedures (described below), and experimental procedures. Consultant training included review of written materials, didactic instruction, observing an experienced consultant complete each element of the procedure, completing each procedure while being observed by an experienced consultant or the first author, and co-completing activities with an experienced consultant to assure that consultants completed activities and assessments accurately. Additionally, implementation of the procedures was reviewed with the consultants weekly by the first two authors.

The participants in this study were the first 48 teachers who made a referral, who consented to participate in the study, and for whom the referred student's parent also consented to participate. Forty-five of the 48 teachers completed the study protocol with one teacher from each of the three treatment conditions dropping out prior to completing the study. These three cases were not completed due to the student moving.

The teachers were predominantly female (44 female and 1 male) and the majority of them had received a permanent teaching certificate (80%). The teachers' years of teaching experience ranged from 0 to 35 years experience with a mean of 9.5 years. ANOVA and Z2 did not reveal any statistically significant differences between treatment groups on years of teaching experience, certification status, or gender with α_pc = .05.

The students who completed the study included 32 boys and 13 girls. They were enrolled in general education, kindergarten through fifth grade, and the mean grade level was 2.6 (treating kindergarten as Grade 0). Students were referred for academic skill problems ( 18), challenging behavior ( 7), academic work habits ( 2), and challenging behaviors plus work habits ( 18). The distribution of target concerns across conditions is presented Table 1. ANOVA (grade) and Z2 (target concern) did not reveal any statistically significant differences between treatment groups on student grade level or referral concern with α_pc = .05. The χ² was significant (χ² = 6.7, df = 2, p = .035) for gender, with girls being disproportionately represented in the performance feedback condition (described below). Preliminary ANOVAs found no differences in student outcomes for either teacher rated behavior change of change in directly observed behavior based on gender with a at .05. Preliminary analyses were conducted to determine whether student outcomes differed by type of concern (behavioral versus academic). A between groups t-test for the primary behavioral outcome measure (described below) indicated no significant difference (t = 1.1, df = 39, p = .28). An analysis of covariance for teacher rating of student outcome (described below), with pretreatment rating as a covariate, also indicate the absence of a significant effect for type of referral concern (F = .07, df = 1, 42, p = .79).

The study was conducted at six urban elementary schools in the Southeastern United States. All of the schools included a prekindergarten through fifth grade or kindergarten through fifth grade configuration. The schools' populations were nearly exclusively African American (96%). A high level of poverty in these schools is suggested by data indicating that 90% of the student population was receiving either a free school lunch or lunch at a reduced price. In most instances (86% of enrollment), the student received lunch at no charge.

Treatment integrity. The primary dependent measure for this study was the extent to which teachers implemented the students' intervention plans as they were designed, as assessed by permanent products of the intervention. An intervention plan was developed for each student in collaboration with that student's teacher within the Behavioral Consultation (BC) framework (Bergan & Kratochwill, 1990). The procedures for developing intervention plans are described below. Although intervention plans were specific to each student's needs, all intervention plans included monitoring target behaviors (e.g., acts of aggression and/or oral reading fluency) and providing rewards contingent upon attaining a goal. All intervention plans included completing activities that produced permanent products such as tutoring work sheets, student self-monitoring records, or records of teachers' monitoring of student behavior. Similar to previous research in this area (see Mortenson & Witt, 1998; Noell, Duhon et al., 2002; Witt et al., 1997), plan implementation was assessed by permanent products. Measurement by permanent products is a practical strategy for the assessment of behaviors that occur over the course of an entire school day. Additionally, assessment by permanent products is unlikely to create reactivity to measurement. Each consultant devised a permanent product checklist prior to the intervention beginning for his or her cases to permit scoring integrity. The approach to deriving permanent product lists for interventions was modeled on previous research in this area (see Mortenson & Witt, 1998; Noell et al., 1997; Noell et al., 2000; Noell, Duhon et al., 2002; Witt et al., 1997). If interventions included elements that consultants were unsure how to include in the assessment plan these issues were reviewed with the first author.

For each intervention plan the number of permanent products that would occur each day if the plan were implemented as designed was identified. A score was derived for each day by counting the number of accurately completed products and dividing by the total number of products. For example, if a student was using a self-monitoring record, the records were examined to determine whether teacher agreement with student recording was recorded when it was planned to occur, if the results of the day were tallied correctly, if the teacher correctly recorded when the student met his or her goal, and whether the teacher recorded providing or withholding the reward accurately. In some instances a permanent product was the result of a teacher's report of his or her own behavior, such as providing a reward. However, the accuracy of that action was scored based upon the child behavior permanent products. Similarly, for a peer-tutoring intervention, products were scored for evidence that the students had indeed had the session, that the teacher checked the work accurately, and that the teacher implemented the reward criterion accurately. For detailed descriptions of this process for academic and behavioral interventions please see Noell, Duhon et al. (2002), Noell et al. (2000), and Noell et al. (1997).

At the end of the 3-week intervention trial, all of the intervention materials were collected and scored by the consultant using the implementation checklist that had been developed when the intervention plan was developed. An example of a permanent product checklist for an academic intervention and a behavioral intervention are available from the first author.

Student outcomes. A direct behavioral measure of the target concern was collected by a member of the research team prior to and at the conclusion of the intervention trial to directly assess the degree of student behavior change. In two instances the behavioral measure was collected by the student's teacher using a behavioral event log because the behavior was a low frequency (0 to several instances per day), high intensity behavioral concern. Behavioral data collected by consultants and teachers permitted examination of student behavior change in relationship to treatment condition, treatment integrity, and teacher ratings of behavior change. For academic concerns, the behavioral data was performance on the target task under standardized conditions such as curriculum-based measurement (CBM; Shinn, 1989). For social behavior concerns that occurred at a moderate to high rate, assessments consisted of 30-minute direct observations in the classroom using 10-s interval recording. Problematic behaviors were recorded using partial intervals; desirable behaviors were recorded using whole intervals. For low incidence, high intensity problem behaviors, a week-long behavioral incident log was used. The teacher maintained the behavioral incident log for the week prior to initiation of the intervention plan and for the last week of intervention. The teacher recorded the number of occurrences of the target behavior each day. A direct measurement of each target concern was obtained for students with more than one target concern.

To create a standardized estimate of student behavior change across the diverse concerns, a student behavior change index (SBCI) was calculated. The SBCI was developed to allow direct observational assessment data across diverse behaviors to be summarized on a common metric (percentage change of baseline levels). For behaviors for which the goal was behavior decrease, the SBCI was calculated as the value of the pretreatment observation minus the posttreatment observation with this sum divided by the value of the pretreatment observation. The resulting value reflects the percentage change in student behavior from pretreatment to post treatment as a percentage of baseline levels. If a student exhibited six episodes of aggression during pretreatment observation and three at posttreatment observation, this reflected a 50% decrease in aggression: [(6-3)/6 x 100%]. The order of the subtraction was reversed for concerns for which the goal was to increase behavior (e.g., reading). If a student read 40 words per minute (WPM) at pretest and 70 WPM at posttest that would result in a 75% increase [(70-40)/40 x 100% = 75%].

Although the SBCI permits summarizing observational assessment across diverse concerns, it has two major limitations. First, as is true with any single point pre/post assessment, it does not capture the variability in behavior that would permit an effect size estimate. Second, a research history examining the use of the SBCI and its psychometric properties does not exist. This is particularly relevant as it relates to reliability issues associated with difference scores. Despite these limitations, the authors chose to include the SBCI in the current study as a practical means of incorporating direct observational assessments across diverse behaviors into the evaluation of consultation services. The exclusive reliance on rating scales appeared to be a less ideal approach.

Teachers rate a the severity, manageability, and tolerability of the students' referral concerns prior to and at the conclusion of the intervention. This dependent variable was collected to assess the relationship between teacher perceptions of student functioning as it related to treatment condition, treatment integrity, and student behavior change. This assessment was modeled after Fuchs and Fuchs (1989), who developed it based on Safran and Safran (1985). This rating permitted teachers to provide their assessment of the specific student behaviors that had been targeted for intervention. Teachers rate a each aspect of the problem on a 1 to 5 Likert rating scale with 1 corresponding to mild severity, easily tolerated, and easily manageable and 5 corresponding to most severe, intolerable, and unmanageable.

Teacher process perceptions. Each teacher completed the Intervention Rating Profile--15 (IRP-15; Martens, Witt, Elliott, & Darveaux, 1985) prior to and at the conclusion of the intervention trial. The IRP-15 was collected to examine the influence of acceptability on implementation, the influence of intervention use on acceptability, and to assess the extent to which the interventions that were developed were of similar acceptability across conditions. The IRP-15 asks teachers to rate 15 evaluative statements regarding the extent to which a treatment is acceptable, appropriate, and likely to be efficacious. The IRP-15 has been reported to have high internal consistency (Cronbach's α = .98) and to have high validity coefficients with related measures (r = .86; Witt & Martens, 1983).

Teachers also completed the Consultant Rating Profile (CRP), which was developed specifically for this study. The CRP contains 10 items that are rate a on a 7-point Likert scale from 1 (strongly disagree) to 7 (strongly agree). The CRP is available from the first author. The first seven items ask the teacher to rate the extent to which the consultant was helpful and that consultation was an effective use of their time. The last three questions ask about the extent to which the intervention was implemented as planned, was effective, and the teacher was satisfied with the intervention's effects. This instrument was collected to examine how different follow-up procedures may influence teachers' evaluations of the extent to which consultants were helpful and their perception of whether interventions are implemented as designed and are beneficial. A secondary point of interest was the concordance between teacher-reported implementation and direct assessment of the same.

The consultation process generally conformed to the model that has previously been described as BC (Bergan & Kratochwill, 1990). Following receipt of the referral, the case was assigned to a consultant who completed a Problem Identification Interview (PII). The PII was organized around a semistructured interview form (available from the first author). The PII focused on identification of the student's target concerns in overt behavioral terms, identification of alternative acceptable behaviors, environmental events surrounding the problem behaviors, and other general background information relevant to the current concern. The interview concluded with the consultant scheduling direct assessments of the student's target concern(s).

Following the interview, the consultant conducted a direct assessment of the student's concerns. This differs from most traditional descriptions of BC that typically emphasize teacher data gathering at this stage of the process. The consultant observed all students in the classroom during one or more times that were the focus of concern. In addition, for all students for whom academic skill deficits were suspected or were the focus of the referral, a direct assessment of academic skills was conducted. The assessment emphasized the use of curriculum-based assessment (Shapiro, 1996) and comparison to instructional placement standards. In some instances procedures were adapted to assess specific academic concerns for which standardized CBM procedures had not been developed. All students who received an academic assessment also completed a skill versus a performance problem assessment (see Noell, Freeland, Witt, & Gansle, 2001, for greater detail). This assessment required repeating the academic assessment with a reward contingency in place for improved performance. The intent of this assessment was to determine whether the student's poor performance was the result of poor academic skills or poor motivation. If a student performed well when performance led to a reward such as a soft drink, the consultant hypothesized the poor performance was the result of poor motivation rather than poor skills. If the student performed poorly even when improved performance would lead to a highly desirable reward, a skill deficit hypothesis was retained.

Following the direct assessment of the student, the consultant met with the teacher to complete the Problem Analysis Interview (PAI). During this meeting the consultant and teacher discussed the assessment results and developed an intervention plan. Intervention plans for challenging behavior concerns emphasized monitoring behaviors, setting goals, and providing systematic rewards for goal attainment. Each behavior intervention plan was developed considering the students' developmental level, any suspected behavioral function, the student's reinforcer preferences, the pretreatment frequency of the behavior, desired behaviors, and acceptable levels of the behaviors. Interventions were developed such that consequences corresponded to student preferences and/or hypothesized functions of behavior. Behavior monitoring periods were generally shorter and rewards were more frequent for younger students. Goal levels for behaviors were set collaboratively by the consultant and teacher to require improvement in behavior for the student to achieve success, but low enough that initial success appeared probable. Goals were reset if that appeared necessary as the student progressed.

Intervention plans for academic skill deficits provided remedial instruction that included modeling, practice, and feedback. These interventions used pretreatment academic assessment data to identify the appropriate target skill or level of task difficulty by comparing student performance to instructional placement standards or benchmarks. Interventions were designed to maximize student response opportunities within a context such as peer tutoring. Response accuracy feedback was provided either immediately or within the session based on the targeted academic skill. Guided practice or coaching was provided for solving more complex problems such as text search or multiple-step mathematics problems. Intervention plans for academic performance deficits included monitoring academic products, setting goals for work, and providing rewards for goal attainment. Monitoring schemes and goals were based on pretreatment assessments and teacher expectations. Goals were set such that they required substantive improvement, but appeared attainable. The process for devising academic interventions generally followed the process described by Noell (2002).

Once the consultant and teacher had reached agreement regarding the intervention, the consultant prepared or gathered all of the necessary materials for implementation and prepared the intervention plan. The consultant met with the student to review the plan and provide any training the student might need (e.g., how to complete a self-monitoring record). The consultant also met with the teacher and any other person who was contributing to the intervention (e.g., a peer tutor) and taught them how to complete their responsibilities in the intervention plan.

The consultant then observed the initial implementation of the intervention in the classroom. If any problems arose when the intervention was implemented, the consultant and teacher problem solved them at that time. For example, if it was apparent that the number of reward occasions was set impractically high they would be revised downward. Alternatively, if the peer tutor appeared to have forgotten some parts of the peer tutoring procedure they were retrained before proceeding. If the teacher did not implement an element of the treatment plan the consultant pointed out the omitted element and asked the teacher to complete it.

Following this consultation process the teacher implemented the intervention for 3 weeks. This duration was chosen for this study for three reasons. First, previous research has clearly demonstrated that this window is sufficient to observe the primary variable of interest: treatment integrity (Noell et al., 1997; Noell et al., 2000; Noell, Duhon et al., 2002). If treatment implementation is going to deteriorate, it will be very low by that point. Second, for those cases with poor implementation, the authors did not wish to prolong unnecessarily the absence of intervention for referred students if weekly follow-up alone was ineffective. Third, the research cited above has demonstrated measurable improvement for students' functioning within 3 weeks or less when provided direct behavioral intervention.

The experimental design for the primary research question was a 3-by-3 split-plot analysis with factors for time (week of the treatment trial) and condition. All other measures were collected within a 3-by-2 split-plot factorial design. The within participants factor had two levels consisting of pretreatment and posttreatment assessment. The between participants factor had three levels consisting of the three follow-up procedures embedded into the consultation process. Participants were randomly assigned to one of the three follow-up conditions with two constraints. First, the number of participants was equal across conditions. Second, all consultants served in all conditions to as nearly equal an extent as the first constraint and random assignment would permit. For each consultant a randomized order of condition assignments for cases, student-teacher dyads, was generated in blocks of three so that all three conditions appeared in each block. Cases were then placed into conditions based upon the order of referral and the predetermined randomly generated order of assignments. All consultants did serve in all conditions except one who only served on one case. A pre-post design for student and teacher measures (other than treatment integrity) was selected due to concerns regarding reactive effects that would likely result from weekly direct assessments of the student and the teachers' completion of relevant rating scales.

All three conditions involved intervention and a no treatment control was not provided for several reasons. First, the primary dependent variable (treatment integrity) cannot be studied in the absence of an intervention plan. Second, the focus of this study was not whether or not consultation is effective when contrasted with no services, but on how follow-up and social influence moderate implementation and how that, in turn, moderates outcome. A no treatment control group does not contribute to that purpose. Third, in the context of the current study, the weekly follow-up condition (described below) can be conceptualized as a current best standard treatment control against which the other conditions can be compared. Finally, as there was no compelling reason within the study's aims to withhold treatment, and given that previous research suggests BC is effective, it was evaluated as ethically questionable to withhold treatment.

Weekly. Weekly follow-up was conceptualized as corresponding to a current best practice approach to consulting practice in schools.

Weekly follow-up consisted of a brief follow-up meeting between the consultant and the teacher that was structured as an abbreviated Plan Evaluation Interview (PEI). The follow-up contact was structured around a brief interview in which the consultant asked about the extent to which the plan was implemented that week, the extent to which the student was improving, and if the teacher had any questions or concerns. Materials were not reviewed unless the teacher asked that the consultant look at them and no treatment integrity information was shared with the teacher.

Commitment emphasis. The commitment emphasis (CE) condition included all of the elements of weekly follow-up. In addition, a social influence procedure was introduced at the final meeting with the teacher prior to intervention implementation. The social influence procedure consisted of reviewing five specific points with the teachers that were designed to enhance the correspondence between their commitment to implement the intervention and actual implementation. First, the consultant described how people frequently make commitments to behavior change, but fail to follow through with that plan due to other time demands. The consultant then discussed the importance of the intervention plan as a commitment to the student and his or her parents. Third, the consultant discussed the loss of credibility that would accompany failure to keep the commitment and the possibility that not following through could do harm. Fourth, the consultant discussed the importance of implementing the intervention to evaluate its effectiveness and to keep the commitment made to the child. Finally, the consultant and teacher discussed proactive steps the teacher could choose to take to support implementation such as goal setting, self-monitoring, and self-rewards for implementation. Teachers were free to devise any strategy they thought would be helpful or no strategy at all.

Performance feedback. The performance feedback (PFB) procedure used in this study was modeled after the previous research examining PFB as a means of enhancing treatment plan implementation (Mortenson & Witt, 1998; Noell et al., 2000). PFB consisted of meeting briefly with the teacher, reviewing the intervention permanent products, graphing student behavior, and graphing intervention implementation. Although the permanent products were reviewed with the teacher, the teacher retained them until the end of the treatment trial as was the case in the other two conditions. The student behavior graph was based on the behavior monitoring that was integral to the intervention. The consultant provided positive feedback regarding steps that were completed and identified steps that were omitted or implemented incorrectly. The consultant and teacher then discussed the importance of any missed steps, problem solved for future implementation, and scheduled their next contact. In contrast to previous research in this area, the PFB procedure in this study was implemented using a rapid thinning to once per week follow-up. PFB was implemented following the first day of implementation and every day thereafter until the teacher implemented the intervention with 100% integrity. PFB was then implemented every other day until the teacher implemented the intervention with 100% integrity for 2 days. PFB was implemented weekly thereafter. The minimum number of follow-up contacts that would occur in this condition was 4 and this was the number of contacts for 9 cases. The mean number of follow-up contacts was 5.2 with a range of 4 to 6.

Consultants were trained through a process that included reading printed materials, participating in didactic instruction, observing a skilled model implement the procedure, and then implementing procedures while being observed to assure that they had mastered the consultation procedures. The following procedures were used to maintain the integrity of the implementation of the consultation procedures: (a) consultants followed a written procedural guide that described the required activities in a checklists format, (b) detailed procedural checklists were provided for specific activities such as performance feedback or commitment emphasis, (c) consultants completed interview forms and checklists to assure that meeting objectives were completed, (d) consultants recorded the dates when activities were completed and provided the products of those activities (e.g., observation records, interview forms), and (e) all cases were reviewed by the first two authors during weekly supervision meetings to monitor implementation of the consultation protocol. In addition, cases were reviewed to assure that all activities had been completed as planned. In the four instances in which an activity was not completed as planned, follow-up was conducted to determine why that was the case. In all instances the activity could not be completed because the student changed schools or class assignment at about the time the intervention ended preventing a follow-up observation.

For all analyses, condition was a between subjects factor and time was a within subjects factor. The primary outcome measure targeted in this study was the extent to which teachers in each of the three conditions implemented the referred students' intervention plans. An analysis of variance (ANOVA) was conducted examining effects for time, condition, and the interaction of time and condition. For this analysis, time consisted of three levels that were the mean treatment plan implementation for each of the 3 weeks. Condition contained three levels, consisting of the three follow-up conditions (Weekly, CE, and PFB). Alpha was set at .05 for each main effect. ANOVA did not reveal a significant main effect for the interaction of time and condition.

ANOVA indicated a significant main effect for condition [F( 2, 42) = 9.0, p = .001] with a large effect size η_p² = .81). To protect the family-wise error rate, this significant main effect was probed using the Tukey honestly significant difference (HSD) procedure for all simple pairwise comparisons. PFB differed from CE and Weekly, which were not statistically significantly different from one another. PFB was associated with substantially higher levels of treatment integrity than the other two conditions. ANOVA also revealed a statistically significant main effect for time (F( 2, 42) = 10.0, p < .001) with a small effect size (η_p² = .25). This effect was probed with dependent t-tests with α_pc = .05/3 = .017. Treatment integrity the first week was statistically significantly different from integrity for Weeks 2 (t = 3.4, df = 44, p = .001) and 3 (t = 3.7, df = 44, p = .001), which did not differ from one another (t = 1.1, df= 44, p = .258). Treatment integrity was somewhat higher the first week of plan implementation across all of the consultation conditions. Treatment integrity for each week by treatment condition is presented in Table 2 and in Figure 1.

Teachers also rated the extent to which they perceived the intervention as having been implemented as planned. A one-way ANOVA for condition did not indicate any statistically significant differences between groups in the degree to which they perceived the intervention as having been implemented. Generally, teachers rated implementation as high M = 6.5 on a 1- to 7-point Likert rating scale (see Table 3).

A one-way ANOVA for condition revealed statistically significant differences between conditions [F( 2, 38) = 10.7, p < .001] for the student behavior change index (SBCI) yielding an effect size of η_p²= .36. Post-hoc comparisons employing the Tukey HSD revealed that the degree of behavior change for the PFB group differed from the other two groups, which did not differ from one another. The PFB group exhibited substantially greater student behavior change than either of the other groups. The mean SBCI for each condition is presented in Table 4.

Teachers rated the severity, manageability, and tolerability of the students' target concerns prior to and at the conclusion of the treatment trial. An internal consistency analysis suggested that these three items could be treated as a single scale (Cronbach α = .85 for the pretreatment assessment and .91 for the posttreatment assessment). An ANOVA was conducted examining the two assessment occasions (time), the treatment conditions, and the interaction term. ANOVA was not significant for the interaction of time and condition. The main effect for times was significant [F( 1, 42) = 11.6, p = .001] and its effect size was small (η_p²= .21). This effect did not require any further probe as it is a comparison of two means and it suggests the posttreatment mean was lower than the pretreatment mean. ANOVA revealed significant main effects for condition [F( 2, 42) - 5.7, p = .007] with a small effect size (η_p²= .21). To follow up the significant between group differences, the Tukey HSD was conducted. It revealed significant differences between the CE group and Weekly groups. None of the remaining comparisons was statistically significant. Mean pre- and postratings are presented in Table 4.

The extent to which teachers initially perceived the interventions as acceptable was assessed with the IRP-15 administered prior to beginning the intervention. Additionally, the extent to which use of the interventions influenced the acceptability of the interventions was assessed with readministration of the IRP-15 at the end of the treatment trial. A two-way ANOVA for condition, time, or the interaction of condition and time revealed no statistically significant effects. Generally, teachers perceived interventions as highly acceptable as indicated by a mean rating of 5.1 on a 6-point scale across conditions for both the pretest and posttest. The mean rating for the Weekly condition was 5.3, for CE was 5.1, and for PFB was 5.0.

The CRP was used to assess the teachers' perceptions of the consultants with whom they worked and of treatment outcomes. The CRP was conceptualized as containing a seven-item scale relevant to the consultant and then distinct items related to treatment integrity, treatment efficacy, and satisfaction with treatment efficacy. A preliminary internal consistency analysis suggested that the seven consultant items could be treated as a single scale (Cronbach's α = .89). A one-way ANOVA across conditions revealed no statistically significant differences for the consultant rating. Overall, consultants were rated positively with a mean rating of 6.5 on a 7-point scale across conditions. ANOVA (one-way across conditions) for the treatment integrity, teacher perception of effectiveness, and teacher satisfaction ratings revealed no significant effects for condition. Descriptive data for the CRP are provided in Table 3.

Due to their conceptual importance to the BC literature, the relationships between several variables were examined. Due to the relatively small sample size and the number of correlations examined (seven), the risk of both Type I and Type II errors is relatively high. However, given the extensive discussion of these relationships in the literature and the general lack of data describing the strength of these relationships, the authors argue that their descriptive value outweighs the risks that arise in dichotomous statistical probability decision making.

The first relationship examined was the correlation between the pretreatment rating of the intervention's acceptability (IRP-15) and the mean integrity score for the 3 weeks. The relatively strong relationship between acceptability and implementation that has been hypothesized by a number of authors (see Eckert & Hintze, 2000 for a discussion of multiple models) did not emerge (r =. 13, n = 45, p = .38). The remaining correlations between CRP, SBCI, total integrity, and change in teacher rating of the target concern are presented in Table 5. The most notable finding from these exploratory analyses was the moderate (r = .44, n = 41, p = .006) relationship between treatment integrity and student behavior change. Although the relationships between teacher ratings of problem behaviors, CRP, SBCI, and treatment integrity were not statistically significant, future research that is designed to examine these relationships may yet prove to be a productive line of inquiry.

This study provides further support for the efficacy of PFB as a follow-up procedure for BC that improves treatment implementation. When PFB was provided, the level of implementation was substantially superior to the other two consultation procedures examined. The effect size for follow-up condition on treatment integrity is within the range that has typically been described as a large effect. This study extends the literature by providing the first randomized field trial examining implementation of PFB targeting treatment implementation in BC. This study also extends the literature by demonstrating that implementation of interventions began at relatively low levels and deteriorated to very low levels by the third week of implementation (M = 23.1%) for the group receiving weekly follow-up. This is an important extension of the literature for two reasons. First, the previous literature demonstrated deterioration of implementation in the absence of follow-up contacts over periods ranging from 3 to 16 days (e.g., Noell et al., 1997). This absence of frequent follow-up is consistent with some descriptions of BC that call for follow-up contact at a relatively distal point in time (e.g., 4 weeks; Galloway & Sheridan, 1994). Second, the poor and deteriorating performance of the Weekly follow-up group demonstrates that contact alone, even structured outcome and implementation focused contact, was not sufficient to maintain implementation. Simply meeting and talking about implementation was not enough to support implementation. Review of implementation data appears to be a critical factor in maintaining implementation.

The data for CE are more difficult to interpret with confidence. Examining the means for treatment implementation and the SBCI would suggest that CE was superior to weekly follow-up, but these apparent differences were not statistically significant. This may be a statistical power issue owing to the relatively large within group variability and the modest n per cell. The most that can be concluded about the types of social influence strategies that were examined in this study is that they warrant further investigation. Future studies that refine the procedures to make the social influence procedure produce a larger and more consistent effect or study them using a larger sample may shed greater light on their utility. However, if large samples are needed to detect an effect, it may become statistically significant without becoming educationally, clinically, or socially significant. The more profitable direction for future research would appear to be either incorporating additional antecedent adult behavior change strategies such as self-management (e.g., Bandura, 1991) or integrating self-management and PFB.

Although the behavioral measures for student outcomes, SBCI, and treatment integrity suggest that PFB was associated with substantially superior outcomes to Weekly or CE, the teacher rating data provide a different picture. On average, consultants were perceived very positively and similarly across conditions based on the CRP. Furthermore, teacher ratings of treatment integrity, intervention effectiveness, and satisfaction with the intervention effects were very similar across groups. The IRP- 15 ratings suggest that interventions were nearly uniformly perceived as acceptable across conditions prior to and after the implementation period. Taken as a whole, it would appear that teachers may have perceived the consultation process similarly and positively regardless of condition. Also, similar to previous consultation research, concordance between direct measures of student behavior change and teacher ratings was not high (Fuchs & Fuchs, 1989). This may be partially a result of the brevity of the intervention. Behavioral change may not yet have been enduring or large enough to change teachers' perceptions.

Collectively, the differences between the direct assessments of integrity and child behavior versus the collection of teacher ratings suggests a cautionary note for evaluating the large number of studies that have relied on teacher ratings as the primary or sole dependent measures. Teachers uniformly reported high levels of satisfaction and treatment integrity; however, more direct assessment suggested that implementation and student behavior change varied substantially across conditions with poor outcomes for the Weekly follow-up group and high levels for the performance feedback group. These data demonstrate that consultation procedures that are effective when assessed by teacher ratings are not necessarily effective in producing intervention implementation or child behavior change. Somewhat surprisingly, teachers in the weekly follow-up group appeared to be satisfied with a consultation process that led to little intervention implementation or student behavior change.

This study also provides descriptive data regarding several classic theoretical issues related to consultation that have received scant empirical scrutiny to date. The correlation between child outcomes as assessed by the SBCI and treatment integrity was in the moderate range. This level of correlation is consistent with previous meta-analytic research (r = .51; Gresham, Gansle, Noell, Cohen, & Rosenblum, 1993), and is not surprising when one considers the attenuation that is likely to result from the limitations of the child behavior measure (a single point in time assessment) and intervening variables such as variations in resistance to treatment across children. The low level of correlation between treatment acceptability and treatment implementation is inconsistent with the commonly advanced theoretical models for treatment integrity and BC (see Eckert & Hintze, 2000). The generally high level of acceptability would naturally attenuate any correlation, but this does not completely disqualify these data from addressing the hypothesized positive relationship between acceptability and treatment implementation. Teachers in the Weekly group did not implement treatments to a substantial degree, despite having rated those treatments as acceptable. This suggests that acceptability is not sufficient to assure implementation. Although it appears reasonable that some level of acceptability may be necessary for treatment implementation, that is an issue this study was not designed to address. Future research that prescribes treatments over a wider range of acceptability would be in a better position to examine the necessity of acceptability to achieve implementation. These data also do not provide support for the hypothesis that intervention use would lead to increased acceptability. However, this may be partially the result of ceiling effects with treatment being initially rated quite favorably.

Several cautionary notes are worth considering in evaluating the generality and strength of these findings. First, the sample of consultees was relatively small and was somewhat homogenous (urban teachers in a Southeastern city). The extent to which parents or teachers in a different ecology would respond similarly to this consultation model is unknown. A second and related issue is that the consultants were doctoral students in school psychology rather than practicing school psychologists. Third, interobserver agreement or scorer reliability data were not collected.

The reliance on permanent products to assess intervention implementation rather than direct observations is another potential limitation. It is possible for a teacher to complete products without implementing the intervention. Although it is possible, it appears unlikely. The most obvious indicator that this did not happen is the juvenile character of the writing for the student-completed elements of interventions. However, future research could examine the agreement between direct observation and measurement by permanent products for intervention implementation. An additional limitation of the study is that the integrity of the consultation process was not directly measured by audio recording, video recording, of a live observer.

An additional limitation was that the PFB group had more frequent contact with the teachers in follow-up than did the other two conditions. The reason for this is that PFB was modeled after the previous PFB research, with a rapid fading to weekly follow-up. Future research is needed to assess the extent to which PFB remains effective if it is provided less frequently from the outset at weekly or once per 2-weeks interval. Consultants were asked to record the duration of each meeting. The mean total time in follow-up meetings was 18.7 minutes for Weekly, 11.1 minutes for CE, and 30.5 minutes for PFB. It is interesting to note that the ordering of the amount following up contact do not match the order of the mean treatment integrity.

Although positive results were obtained for the participants in the PFB group, three weeks is a relatively short duration for treatment implementation. This treatment trial period was long enough to observe the primary phenomenon of interest, treatment integrity, but was not sufficient to examine the durability of the PFB procedure over a longer treatment trial that may be more reflective of common practice. Finally, the number of participants per cell was relatively small ( 15). However, this number greatly exceeds the number of participants in previous published studies examining PFB and treatment integrity and is consistent with the very limited number of group experimental studies examining consultation variables in schools (e.g., Fuchs & Fuchs, 1989; Fuchs, Fuchs, & Bahr, 1990; Fuchs, Fuchs, Bahr, Fernstrom, & Stecker, 1990). The large effect obtained for condition on treatment integrity and the moderate correlation between treatment integrity and child outcome across conditions suggests that this n was sufficient for this study's primary aims.

These results suggest several possible lines for future research. For example, would a procedure combining the CE procedure with the PFB produce results that were superior to PFB alone? Similarly, future research could systematically manipulate the acceptability of treatments to examine what level of acceptability is sufficiently low that it would influence implementation. It may be the case that once acceptability reaches that point, consultees may simply refuse the treatment. If that is the case, acceptability may only be of importance in the initial process of obtaining a commitment. Additionally, research examining means of strengthening the CE procedures is a viable direction for future study. Finally, study of the adaptation of these procedures to parents or other care providers is an obvious direction for additional research.

The most important findings of this study are consistent with previous research in this area (e.g., Mortenson & Witt, 1998; Noell et al., 2000; Noell, Duhon et al., 2002; Witt et al., 1997). Treatment implementation following consultation was frequently poor and commonly deteriorated over time in the absence of performance feedback; however, performance feedback was an effective means of sustaining treatment implementation. This study extends previous research by demonstrating these effects in a randomized field trial that included a rapid fade of the performance feedback procedure to once per week. Additionally, this study suggests that consultants who provide performance feedback are perceived by consultees positively and similarly to those who do not. The data also suggest that teacher ratings provide a more uniformly positive evaluation of consultation than more direct behavioral measures do. The moderate correlation between treatment implementation and child behavioral outcome provides further evidence supporting the importance of attending to treatment integrity. In summary, these data provide empirical support for performance feedback as one method of responding to a pervasive, recalcitrant, and underresearched problem in the psychological and educational treatment of children: assuring treatment plan implementation.

Author Note. This study was supported in part by a research grant from the Society for the Study of School Psychology (SSSP). Natalie Slider is now at the Virginia Beach Beach City Schools, Virginia. James Connell is now at the May Institute, Norwood, MA. Susan Gatti is now at the Louisiana State University Health Sciences Center at Shreveport.

Legend for Chart:

B - Condition Weekly
C - Condition Commitment Emphasis
D - Condition Performance Feedback

          A             B   C   D

Task engagement         1   1   0
Task engagement &   8   6   4
 challenging behavior
Challenging behavior    2   3   2
Academic skills         4   7   9

Legend for Chart:

A - Condition
C - Week 1
D - Week 2
E - Week 3
F - Grand Mean

         A             B     C          D         E         F

Weekly                 M    45.8      36.1      23.1      35.0(a)
                       SD   34.7      29.8      28.4      31.8
Commitment Emphasis    M    65.8      45.6      45.8      52.3
                       SD   25.5      43.2      38.3      38.6
Performance Feedback   M    81.8      74.2      75.2      77.1(b)
                       SD   17.6      25.6      28.7      24.1
Grand Mean             M    64.5(a)   52.0(b)   48.0(b)   54.8
                       SD   30.2      36.8      38.3      35.7

Note. The number of participants for each cell is 15 for each
week by condition cell. Mean treatment integrity for each week
was calculated by summing the daily treatment integrity scores
and dividing by the number of days in that week. Absences were
not included in the mean. Following omnibus ANOVA for main
effects, the Tukey honestly significant difference (HSD) test
was used to test for differences between conditions and weeks
for marginal means. Means in the same grand mean row or column
that have different subscripts differ at p < .05 based on the
comparison.

Legend for Chart:

C - Weekly
D - Commitment Emphasis
E - Performance Feedback
F - GRand Mean

           A               B      C      D      E      F


Consultant effectiveness   M    46.4   46.5   43.9   45.6
                           SD    3.2    4.3    4.2    4.0
Treatment integrity        M     6.7    6.5    6.5    6.5
                           SD     .5    1.1    1.1     .9
Intervention effective     M     5.1    5.6    5.1    5.3
                           SD    1.8    2.0    1.3    1.7
Satisfaction with effect   M     4.9    5.5    5.1    5.2
                           SD    1.9    2.1    1.3    1.8

Note. There were 15 participants per cell resulting in 45
participants for the grand mean. Judgments were made on a 7-point
Likert rating scale. Consultant effectiveness is the sum of seven
items. No statistically significant differences were identified
for the data contained in this table.

Legend for Chart:

C - Teacher Problem Rating SBCI
D - Teacher Problem Rating Pretreatment
E - Teacher Problem Rating Post Treatment
F - Teacher Problem Rating Rating Grand Mean

         A             B     C         D      E      F

Weekly                 M      2%      12.2   11.8   12.0
                       SD    33%       1.9    2.6    2.3
                       n      13        15     15     15
Commitment Emphasis    M     37%      11.3    7.6    9.4
                       SD    38%       2.7    3.7    3.7
                       n      13        15     15     15
Performance Feedback   M     96%      11.7    9.9   10.8
                       SD   102%(b)    2.3    3.5    3.0
                       n      15        15     15     15
Grand Mean             M     54%      11.7    9.8   10.7
                       SD    79%       2.3    3.6    3.2
                       n      41        45     45     45

Note. SBCI = Student Behavior Change Index. Problem ratings
included ratings of severity, tolerability, and manageability
on a 5-point Likert rating scale. The possible range of scores
was 3 to 15. Within the SBCI column, means that have differing
subscripts differ at p < .05 based upon a Tukey honestly
significant difference post-hoc comparison following a
significant one-way between participants ANOVA.

Legend for Chart:

C - CRP
D - SBCI
E - Treatment Integrity

           A               B    C      D      E

SBCI                       r   -.07
                           p   .685
                           n     41
Treatment integrity        r   -.02    .44
                           p   .919   .004
                           n     45     41
Change in problem rating   r    .28    .29    .21
                           p   .067   .067   .157
                           n     45     41     45

Note. CRP = Consultant Rating Profile. SBCI = Student Behavior
Change Index. Treatment integrity is the mean treatment
integrity across the study. Change in problem rating is the
teachers' pretreatment rating of the target concern minus the
posttreatment rating.

GRAPH: Figure 1. Mean treatment integrity by week for each consultation condition.

Alvero, A. M., Bucklin, B. R., & Austin, J. (2001). An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings. Journal of Organizational Behavior Management, 21, 3-29.

Balcazar, F., Hopkins, B. L., & Suarez, Y. (1985). A critical, objective review of performance feedback. Journal of Organizational Behavior Management, 7, 65-89.

Bandura, A. (1991). Social cognitive theory of selfregulatio. Organizational Behavior & Human Decision Process, 50, 248-287.

Bergan, J. R., & Kratochwill, T. R. (1990). Behavioral consultation and therapy. New York: Plenum Press.

DuPaul, G. J., & Eckert, T. L. (1997). The effects of school-based interventions for Attention Deficit Hyperactivity Disorder: A meta-analysis. School Psychology Review, 26, 5-27.

Eckert, T. L., & Hintze, J. M. (2000). Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology. School Psychology Quarterly, 15, 123-148.

Foxx, R. M. (1996). Twenty years of applied behavior analysis in treating the most severe problem behavior: Lessons learned. The Behavior Analyst, 19, 225-236.

Fuchs, D., & Fuchs, L. S. (1989). Exploring effective and efficient prereferral interventions: A component analysis of behavioral consultation. School Psychology Review, 18, 260-283.

Fuchs, D., Fuchs, L. S., & Bahr, M. W. (1990). Mainstream assistance teams: A scientific basis for the art of consultation. Exceptional Children, 57, 128-139.

Fuchs, D., Fuchs, L. S., Bahr, M. W., Fernstrom, P., & Stecker, P. M. (1990). Prereferral intervention: A prescriptive approach. Exceptional Children, 56, 493-513.

Galloway, J., & Sheridan, S. M. (1994). Implementing scientific practices through case studies: Examples using home-school interventions and consultation. Journal of School Psychology, 32, 385-413.

Gansle, K. A., & McMahon, C. M. (1997). Component integrity of teacher intervention management behavior using a student self-monitoring treatment: An experimental analysis. Journal of Behavioral Education, 7, 405-419.

Gonzales, M. H., Aronson, E., & Costanzo, M. A. (1988). Using social cognition and persuasion to promote energy conservation: A quasi-experiment. Journal of Applied Social Psychology, 18, 1049-1066.

Gresham, F. M., Gansle, K. A., Noell, G. H., Cohen, S., & Rosenblum, S. (1993). Treatment integrity of school-based behavioral intervention studies: 1980-1990. School Psychology Review, 22, 254-272.

Greenwood, C. R., Terry, B., Arreaga-Mayer, C., & Finney, R. (1992). The classwide peer tutoring program: Implementation factors moderating students' achievement. Journal of Applied Behavior Analysis, 25, 101-116.

Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821-833.

Howard, D. J. (1995). "Chaining" the use of influence strategies for producing compliance behavior. Journal of Social Behavior and Personality, 10, 169-185.

Jones, K. M., Wickstrom, K. F., & Friman, P. C. (1997). The effects of observational feedback on treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 12, 316-326.

Lipsitz, A., Kallmeyer, K., Ferguson, M., & Abas, A. (1989). Counting on blood donors: Increasing the impact of reminder calls. Journal of Applied Social Psychology, 19, 1057-1067.

Martens, B. K., Hiralall, A. S., & Bradley, T. A. (1997). A note to teacher: Improving student behavior through goal setting and feedback. School Psychology Quarterly, 12, 33-41.

Martens, B. K., Witt, J. C., Elliott, S. N., & Darveaux, D. X. (1985). Teacher judgments concerning the acceptability of school-based interventions. Professional Psychology: Research & Practice, 16, 191-198.

McEvoy, M. A., Shores, R. E., Wehby, J. H., Johnson, S. M., & Fox, J. J. (1990). Special education teachers' implementation of procedures to promote social interaction among children in integrated settings. Education and Training in Mental Retardation, 25, 267-276.

Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247-266.

Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a prereferral academic intervention. School Psychology Review, 27, 613-627.

Noell, G. H. (2002). Direct assessment of clients' instructional needs: Improving academic, social, and emotional outcomes. In M. L. Kelley, D. Reitman, & G. H. Noell (Eds.), Practitioner's guide to empirically based measures of school behavior (pp. 63-84). New York: Kluwer Academic/Plenum Publishers.

Noell, G. H., Duhon, G. J., Gatti, S. L., & Connell, J. E. (2002). Consultation, follow-up, and behavior management intervention implementation in general education. School Psychology Review, 31, 217-234.

Noell, G. H., Freeland, J. T., Witt, J. C., & Gansle, K. A. (2001). Using brief assessments to identify effective interventions for individual students. Journal of School Psychology, 39, 335-355.

Noell, G. H., Gresham, F. M., & Gansle, K. A. (2002). Does treatment integrity matter? A preliminary investigatioon of instructional implementation and mathematics performance. Journal of Behavioral Education, II, 51-67.

Noell, G. H., Witt, J. C., Gilbertson, D. N., Ranier, D. D., & Freeland, J. T. (1997). Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77-88.

Noell, G. H., Witt, J. C., LaFleur, L. H., Mortenson, B. P., Ranier, D. D., & LeVelle, J. (2000). A comparison of two follow-up strategies to increase teacher intervention implementation in general education following consultation. Journal of Applied Behavior Analysis, 33, 271-284.

O'Keefe, D. J., & Medway, F. J. (1997). The application of persuasion research to consultation in school psychology. Journal of School Psychology, 35, 173193.

Peterson, L., Homer, A. L., & Wonderlich, S. A. (1982). The integrity of independent variables in behavior analysis. Journal of Applied Behavior Analysis, 15, 477-492.

Safran, S. P., & Saltan, J. S. (1985). Classroom context and teachers' perceptions of problem behaviors. Journal of Educational Psychology, 77, 20-28.

Shapiro, E. S. (1996). Academic skills problems: Direct assessment and intervention (2nd ed.). New York: Guilford.

Sheridan, S. M., & Gutkin, T. B. (2000). The ecology of school psychology: Examining and changing our paradigm for the 21st century. School Psychology Review, 29, 485-502.

Shinn, M. R. (1989). Curriculum-based measurement: Assessing special children. New York: Guilford.

Swanson, H. L., & Hoskyn, M. (1998). Experimental intervention research on students with learning disabilities: A meta-analysis of treatment outcomes. Review of Educational Research, 68, 277-321.

Vermilyea, B. B., Barlow, D. H., & O'Brien, G. T. (1984). The importance of assessing treatment integrity: An example in the anxiety disorders. Journal of Behavioral Assessment, 6, 1-11.

Weiss, B., & Weisz, J. R. (1995). Relative effectiveness of behavioral versus nonbehavioral child psychotherapy. Journal of Consulting and Clinical Psychology, 63, 317-320.

Weisz, J. R., Weiss, B., Alicke, M. D., & Koltz, M. L. (1987). Effectiveness of psychotherapy with children and adolescents: A meta-analysis for clinicians. Journal of Consulting and Clinical Psychology, 55, 542-549.

Witt, J. C., & Martens, B. K. (1983). Assessing the acceptability of behavioral interventions used in classrooms. Psychology in the Schools, 20, 510-517.

Witt, J. C., Noell, G. H., LaFleur, L. H., & Mortenson, B. P. (1997). Teacher usage of interventions in general education: Measurement and analysis of the independent variable. Journal of Applied Behavior Analysis, 30, 693-696.

~~~~~~~~

By George H. Noell, PhD, Louisiana State University; Joseph C. Witt, PhD, Louisiana State University; Natalie J. Slider, PhD, Louisiana State University; James E. Connell, MA, Louisiana State University; Susan L. Gatti, PhD, Louisiana State University; Kashunda L. Williams, MA, Louisiana State University; Jennifer L. Koenig, MA, Louisiana State University; Jennifer L. Resetar, MA, Louisiana State University and Gary J. Duhon, PhD, Oklahoma State University

George H. Noell, PhD, is an Associate Professor of Psychology at Louisiana State University, Baton Rouge, Louisiana. His research interests are behavioral consultation, intervention implementation, assessment procedures that have treatment utility, and value added assessment of educational programs. His research focuses on high incidence referrals and populations in education. Address correspondence to George H. Noell, 236 Audubon, Department of Psychology, Louisiana State University, Baton Rouge, LA 70803-5501.

Joseph C. Witt, PhD, is a Professor of Psychology at Louisiana State University and Director of the School Psychology program at LSU. His research interests focus on developing programs that provide quality educational experiences for all children and removing barriers to their implementation.

Natalie J. Slider, PhD, is a School Psychologist in the school district of Virginia Beach City Public Schools. Her clinical and research interests include early intervention, teacher, parent, and staff training, and applied behavior analysis.

James E. Connell, MA, is currently completing his doctoral degree at Louisiana State University. Presently, he is an Educational Consultant with May Institute where he works with school districts and families on behalf of students with diverse needs. His clinical and research interests include the architectural application of behavior analysis to all natural and applied settings.

Susan Gatti, PhD, is an assistant professor at the LSU Health Sciences Center-Shreveport. Her research interests include resistance to intervention and effective school-based intervention programs.

Kashunda L. Williams, MA, is a doctoral candidate in the School Psychology program at Louisianan State University. Her current research interests include treatment implementation schedules, academic curriculums, and home-school communication.

Jennifer L. Koenig, MA, is a doctoral candidate in the School Psychology program at Louisiana State University. Her clinical and research interests include academic assessment and intervention, teacher implementation, and resistance to intervention.

Jennifer L. Resetar, MA, is a doctoral candidate in the School Psychology program at Louisiana State University. Her clinical and research interests include effective academic interventions, parent training, increasing treatment integrity, functional assessment, and school consultation.

Gary J. Duhon, PhD, is an Assistant Professor of School Psychology at Oklahoma State University. He is a graduate of Louisiana State University in 2001. His research interests are behavioral consultation, functional assessment of academic deficits, treatment integrity, and targeted interventions at both the individual and group level.