Aboriginal Inquiry: Lifting All Learners

An Impact Assessment of the Aboriginal Enhancement Schools Network (AESN)

§ 3: Methodology and Methods of Analysis

3.0 Methodology and Methods of Analysis

To top of this section § ↑

Before discussing the details of the data collected for this study, it is important to outline for readers how the data was collected and analyzed so that it becomes clear how the claims of impact have been realized. This section of the report describes this process and the framework used.

As was discussed in the in the introduction to this study, this report summarizes what we are describing as the impacts of the AESN . As a result, the design of our study sought to gather how selected participants in the Network describe and illustrate the nature of this impact by sharing their experiences and stories. It is a qualitative approach; that is to say, the methods used have focused on the lived experiences of members of the Network. A focus on the lived experiences draws upon a constructivist, interpretive paradigm (Denzin & Lincoln, 1994) and creates stories or cases that use thick, deep and rich description as its primary tools of verification. In other words, trustworthiness is created through efforts to paint a complete and detailed description of events, approaches and experiences. In particular, this report uses case studies as its primary means of capturing the operation of the Network and its impacts.

3.1 Case study: Intrinsic and Instrumental

To top of this section § ↑

Case study is not a method but rather a practice of representation of what has been investigated and learned through a careful examination of data. Both single and collective cases can be studies; in this report, single cases are developed (reporting on the specific bounded examples of Prince Rupert and Arrow Lakes) but the larger case study of the Network itself is also discussed using narratives and content analysis (in the section described as Impacts of the Network). Stake (1994) describes cases such as these in several ways, including intrinsic and instrumental (p. 237).

The cases of both Prince Rupert and Arrow Lakes should be considered intrinsic cases, because the focus is not on theory per se, but on exploring the details and contexts in which a particular set of events have unfolded. Its purpose is to understand more deeply how the specifics lead to particular outcomes. The later section of the report described as Impacts of the Network is an example of the instrumental case. Here the focus is on how the details of individuals’ reported experiences exemplify the theories and by extension, the implications of these theories to practice. Having said these differences exist, in practice often both theory and deep descriptions of events or stories overlap in the telling of the case or story. A third kind of approach to a case study is evident in the section called AESN Case Study Assessment. In this section we summarize how a group of documents, specifically a set of AESN reports written by members between 2009-2012 constitutes a ‘case’ for seeing how the impact on student learning has been documented over time.

An important outcome of case study is that it can be seen “as a small step toward grand generalization”(Stake, 1994, p. 238). We believe that the scope and range of cases included in this report provide us with the potential for making at least some tentative generalizations about the impact of the Network.

3.2 Impact assessment

To top of this section § ↑

Case study is our primary research approach for demonstrating the impact of the AESN . But what is impact assessment? Recent trends in the social sciences literature document a shift towards impact assessment rather than evaluation measures, particularly for organizations which seek to make social impacts upon its participating community. Such thinking has emerged as organizations have considered how to measure their quality, success and/or the significance or importance of the work to a community or a group, as well as how potential lessons might be derived from the activity of measuring performance. In other words, it is more of a value driven exercise than one driven by quantitative, final outcome measures (Stufflebeam, 2007). Marula et al (2003) suggest impact assessment is better described as “… analogous to a reflective process through which social change actors and advocates articulate their change goals and formulate the criteria with which they will evaluate the successes and failures of change efforts. This in turn guides the actors in rethinking their change efforts, influencing whether and how their further efforts should be modified” (p. 58 as cited in Lall, 2011, p. 5).

3.3 Culturally responsive assessment practices used in this report

To top of this section § ↑

As noted in the introduction, our approach to impact assessment includes what we describe as a “culturally inclusive impact assessment”. In saying this, what we are attempting to demonstrate is a respect for and valuing of traditional or “wise ways” (Halbert & Kaser, 2012) through the design and reporting of the AESN assessment.

3.3.1 A culturally inclusive advisory board

To top of this section § ↑

The Principals of the AESN , along with their colleagues, Dr. Trish Rosborough and Sarah Cormode of the BC Ministry of Education Aboriginal Education Branch, initiated this study through their application to the federal government. As a part of their proposed assessment of the Network, they invited a group of key leaders from around the province to serve as an advisory group. Members of this group included: Dr. Paige Fisher (Vancouver Island University); Sarah Cormode (BC Ministry of Education, Aboriginal Education Branch); Debbie Leighton-Stephens (District Principal, Prince Rupert School District); Dr. Trish Rosborough (BC Ministry of Education); Gloria Raphael (Surrey School District, Aboriginal Education); Laura Tait (District Principal of Aboriginal Education, Nanaimo); Michelle Miller-Gauthier (Network leader, Vanderhoof); Terry Taylor (District Principal of Learning, Arrow Lakes); Donna Weaving (NOII /AESN program staff); Jo-Anne Chrona (Surrey School District). The work of these individuals was to guide the study design and to provide support to the primary researcher during the data gathering and analysis stages of the impact assessment.

3.3.2 Respecting ownership, knowledge and community protocols

To top of this section § ↑

The primary researcher in this study was asked to ensure that any data collected for this study, including any resources or stories shared, followed local protocols—in other words, the appropriate cultural protocol of acknowledging and recognizing knowledge that were the property of particular individuals or clans be sought and respected. This process was built into all study protocols and documents.

3.3.3 Culturally inclusive measures: Using story to share impact

To top of this section § ↑

An approach that acknowledges the centrality of story as a culturally inclusive means of describing impact seemed a powerful and compelling way to meet our goals of reporting on the impacts of the AES Network , and was very much in keeping with the AESN purposes of broadening the knowledge of non-Aboriginal peoples about First Nations histories, cultures and contributions to Canadian society. As a result, the design of our study sought to gather impact stories in a number of different ways. One of these ways was to insert narratives collected from participants and written independent of the researcher observations. These intact narratives are designed to provide exemplary voices that corroborate and extend the observations and analyses of the researchers.

Narratives are also inserted throughout the cases included in the report; these narratives have been extracted from the discussions held with the more than 50 participants in the complete study. As much as possible, we have used the exact words of the participants as this, we believe, is respectful of their voice. However, we also occasionally add in some words to help make stronger connections or provide greater clarity. The oral record is supplemented to enhance meaning making for the reader while ensuring the intentions of the speaker are honoured.

3.4 Timeline for compiling the cases and completing the study

To top of this section § ↑

January 16, 2013: The primary researcher Dr. Catherine McGregor and her assistant, Allyson Fleming, met with the AESN advisory board in Vancouver, BC . At this time, the research methods proposed by the researcher were reviewed by the Board. As a researcher employed by the University of Victoria, the U Vic ethics application was used as the appropriate vehicle for approval. During this initial board meeting the overall research plan was approved. This included a description of the type of research to be completed (case study of selected districts, case analysis of selected AESN case studies, interviews, focus groups and literature analysis). The use of appropriate Aboriginal protocols was discussed to provide guidance to the researchers. The Board also considered and then selected two school districts from those involved in the AESN to be used as case studies for an in depth look at impact of the AESN at the school, district and community level.

January 21, 2013: Letters were sent to the two selected school districts, Arrow Lakes and Prince Rupert, seeking permission to conduct the case studies. Approvals were received on January 23 and January 30th respectively.

January 24, 2013: University of Victoria research ethics approval received.

February 2013: Throughout the month of February the researchers visited various communities throughout BC in order to complete the two case studies as well as conduct interviews and focus groups.

February-March, 2013: The researchers wrote the first draft of the report. Intrinsic case studies, once completed, were circulated to those district participants who could review each for accuracy. The full advisory group was also asked to serve as readers and to verify the accuracy of the draft written text. All participants were given the opportunity to read and review the report and as per U Vic ethics, the opportunity to revise, remove or alter their comments, should they wish to do so.

March 2013: The final draft report was sent to Pink Sheep Media to put into publishable form.

3.5 Summary of data collected

To top of this section § ↑

In this section we describe the different data collection methods used in this study.

3.5.1 Focus groups

To top of this section § ↑

Focus groups were conducted in the two selected school districts. The first focus group was held in Prince Rupert on February 4th. The focus group with the Arrow lakes school district was held on February 26th. Focus groups were also held in: Prince George on February 20th; Vanderhoof on February 18th, and Nanaimo on February 13th. A total of 43 individuals participated in these focus groups.

All participants were provided consent forms and the list of possible questions prior to the focus group being held. When each focus group was held, the researcher reviewed the ethics form and provided another copy for participants to sign indicating their willingness to participate. In cases where permission was given to take photographs, these options were completed on the consent forms.

All focus groups were audio taped. The researcher and research assistant both took notes as the focus group was conducted. Each transcript was reviewed by the researcher and assistant individually and then shared to ensure accuracy.

3.5.2 Interviews

To top of this section § ↑

The AESN principals sent out a group email to all members of the AESN asking them to consider participating in the research study. This email outlined the various ways in which the participants could participate: write a personal narrative about their involvement in the AESN ; participate in an individual interview or participate in a focus group. Those individuals interested in participating sent their contact information to the AESN staff person, Donna Weaving. From the complete list of volunteers, a total of 15 individuals were selected by Drs. Halbert and Kaser to be contacted by the researcher for interviews. Of the 15 individuals contacted, a total of 12 interviews were completed; three individuals did not respond to the request.

Each individual was provided with the suggested questions to be discussed at the interview and a copy of the consent form via email. Mutually convenient times were set up to conduct the interviews; some were conducted face to face, but the majority were conducted via telephone. In each case the researcher confirmed orally that the individual was willing to consent to the interview. All interviews were audio recorded and the interviewer took notes as the interview was held. Interviewees were told that there would not be a transcription of the actual interview produced or circulated due to the time limits of the research. All participants agreed with this protocol.

3.5.3 Written narratives

To top of this section § ↑

Members of the AESN were also encouraged to write individual narrative reflections of their involvement in the AESN . They were provided with a list of four possible questions to address, and then asked to limit their comments to approximately 500-750 words. Participants were also asked to ensure that their narratives did not identify others or make references that might reveal specific identities. Written narratives could be provided with names or anonymously. These narratives were forwarded to Donna Weaving at the AESN office. A total of 11 narratives were received.

3.5.4 Phase 4

To top of this section § ↑

The complete draft report was circulated to the AESN advisory board for review. A final edit followed this phase prior to printing and publication.

3.6 Data Analysis

To top of this section § ↑

All of the written notes from the focus groups and interviews were reviewed for accuracy and then printed for analysis. The analysis had several phases:

3.6.1 Phase 1

To top of this section § ↑

The first look at the collected data was conducted at an AESN Advisory meeting in Vancouver, held on February 28th. At this time, members of the advisory group were provided with copies of the focus group data. A total of four groups were created to read and comment on the focus group. Individuals were asked to read and comment upon the focus group notes they had been given and then to discuss in groups the themes or ideas that seemed to be most important. The entire board then reconvened to share their analysis and thinking about the transcripts they had read. Themes were recorded by the researcher on chart paper. After this discussion, AESN advisory members were asked to list what they considered to be the most important or critical themes that the report should address. This list was created on a white board and later photographed as a record of the group’s thinking.

3.6.2 Phase 2

To top of this section § ↑

After completing focus groups in both Prince Rupert and Arrow Lakes, the researcher and research assistant began the process of putting together case studies to profile in greater detail the activities of the selected school districts involved in the AESN .

District documents and reports were accessed to provide background and demographic data for the school district. Detailed descriptions of the work being done by the district was also extracted from focus group and individual interview notes. General descriptions were followed by analysis of overall impacts of the AESN in the district; these were organized thematically. Where necessary, details were confirmed with two key contacts: Debbie Leighton-Stephens in Prince Rupert and Terry Taylor in Arrow Lakes. After the details of each case were completed, the cases were circulated to the lead contacts in each district to review for accuracy and subsequently edited. As much as possible, no names were used, except where appropriate and with permission.

3.6.3 Phase 3

To top of this section § ↑

The researcher and research assistant read individually all of the interview data, coding for dominant themes as they read. They then shared their coding with each other, and created a common set of themes. We combined these themes into the list generated by the AESN advisory board. In total we generated about 44 themes. From this larger list of themes, we created broader headings that represented promising practices in the literature reviewed and that could be used to capture the 44 themes. These broader headings where used to organize our approach to creating the section of the document entitled Network Impact.