About the Author(s)


Chantal Taylor Email
School of Agriculture, Earth and Environmental Sciences, University of KwaZulu-Natal, South Africa

Jessica Cockburn symbol
Department of Environmental Science, Rhodes University, South Africa

Mathieu Rouget symbol
School of Agriculture, Earth and Environmental Sciences, Centre for Invasion Biology, University of KwaZulu-Natal, South Africa

Jayanti Ray-Mukherjee symbol
School of Liberal Studies, Azim Premji University, Bangalore, India

Shomen Mukherjee
School of Liberal Studies, Azim Premji University, Bangalore, India

Rob Slotow symbol
School of Life Sciences, University of Kwazulu-Natal, South Africa

Department of Genetics, Evolution and Environment, University College, London, United Kingdom

Debra Roberts symbol
School of Life Sciences, University of Kwazulu-Natal, South Africa

Sustainable and Resilient City Initiatives Unit, eThekwini Municipality, South Africa

Richard Boon symbol
School of Life Sciences, University of Kwazulu-Natal, South Africa

Environmental Planning and Climate Protection Department, eThekwini Municipality, South Africa

Sean O’Donoghue
School of Life Sciences, University of Kwazulu-Natal, South Africa

Environmental Planning and Climate Protection Department, eThekwini Municipality, South Africa

Errol Douwes symbol
School of Life Sciences, University of Kwazulu-Natal, South Africa

Environmental Planning and Climate Protection Department, eThekwini Municipality, South Africa

Citation


Taylor, C., Cockburn, J., Rouget, M., Ray-Mukherjee, J., Mukherjee, S., Slotow, R., et al., 2016, ‘Evaluating the outcomes and processes of a research-action partnership: The need for continuous reflective evaluation’, Bothalia 46(2), a2154. http://dx.doi.org/10.4102/abc.v46i2.2154

Original Research

Evaluating the outcomes and processes of a research-action partnership: The need for continuous reflective evaluation

Chantal Taylor, Jessica Cockburn, Mathieu Rouget, Jayanti Ray-Mukherjee, Shomen Mukherjee, Rob Slotow, Debra Roberts, Richard Boon, Sean O’Donoghue, Errol Douwes

Received: 16 Aug. 2016; Accepted: 31 Oct. 2016; Published: 03 Dec. 2016

Copyright: © 2016. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: The KwaZulu-Natal Sandstone Sourveld (KZNSS) Research Programme is part of a collaborative, transdisciplinary research partnership between the University of KwaZulu-Natal and the eThekwini Municipality (EM), aimed at bridging the science-policy-practice gap. The research programme focuses on generating knowledge and capacity to support local land-use planning, management and policy development related to biodiversity and climate change issues.

Objectives: The objectives were (1) to describe how a continuous reflective evaluation approach helped to better understand the research programme and its outcomes; and (2) to assess research outputs and outcomes, relevance of outcomes to the requirements of EM, and participants’ perceptions of the programme (both the outcomes and the process).

Methods: The evaluation took a mixed methods approach, combining various quantitative and qualitative methods such as anonymous individual questionnaires, reflective exercises and group reflections.

Results: The KZNSS programme was successful in capacity building and establishing a long-term partnership, but had lower scientific publication output and practice uptake than expected. Participants’ perceptions changed over time, with a decrease in the perceived success of addressing tangible research outcomes, and an increase in the perceived success of collaborative relationships in the partnership.

Conclusion: Transdisciplinary partnerships can be a means of integrating research into policy and practice through knowledge exchange. An important lesson in the early stages of this partnership was to pay attention to the process and not only the outputs. The study highlights the importance of continuous participatory reflection and evaluation in such partnerships.

Introduction

Programme evaluation and evaluation research have received considerable attention recently (e.g. Fazey et al. 2013; Rossi, Lipsey & Freeman 2003). The quality and significance of research programmes are traditionally evaluated against tangible, clearly measured outputs linked directly to the research itself, such as the number of peer-reviewed publications, number of citations, graduate training and other direct deliverables. Such systems of evaluation tend to suit research in well-defined disciplines but are potentially inappropriate for evaluating interdisciplinary or transdisciplinary research. Transdisciplinary (TD) research results tend to compete with the criterion of academic achievement in disciplinary research and are seldom evaluated in terms of their TD contribution (Bergmann et al. 2005). Given the pluralism of disciplines, research paradigms, and stakeholders’ expectations, inter- and transdisciplinary research programmes require a specific approach for evaluation (Klein 2008).

TD research, as defined in the sustainability sciences, is a research approach that addresses societal problems by means of interdisciplinary collaboration, and by transcending the boundary between science and society (Lang et al. 2012). This is achieved through collaboration between researchers and societal actors or practitioners, enabling mutual learning and co-production of knowledge (Hirsch Hadorn et al. 2008; Jahn, Bergmann & Keil 2012; Lang et al. 2012). This approach implies a focus on the implementation of research outputs into policy and practice, and is an attempt to bridge the science-policy-practice gap (Knight et al. 2008; Van Kerkhoff 2014). The evaluation of TD research programmes therefore requires the assessment of both research outputs and practice outcomes (Bergmann et al. 2005); this raises challenges as implementation often takes longer than the lifespan of the research project, and because the various participants might perceive and assess the success of the research and the uptake of knowledge into practice differently (Roux et al. 2010).

Conventional evaluation of research programmes does not include use of research outputs for decision-making and practice. Evaluation is traditionally considered as a once-off activity, usually conducted at the end of the programme (Rossi et al. 2003). A single evaluation at the end of a programme means that the programme will not receive the benefit of continuous evaluation with the potential for amendment, or the opportunity to evaluate the process (i.e. how the program was conducted) in addition to the content or outcomes (i.e. what the program generated) (Dick 2003; Ferreyra & Beard 2007). In inter- or transdisciplinary research, attention needs to be given not only to the outcomes, but also the quality of the process (Klein 2008). Continuous evaluation explicitly addresses learning and accountability, and provides an opportunity for reflection by the participants in the programme (Van Ongevalle, Huyse & Van Petegem 2014).

There are several advantages to internalising evaluation and reflection activities within a team (Van Ongevalle et al. 2014), including that: (1) participants gain more ownership of, and accountability for, the process (Roux et al. 2010); (2) participants can adapt the way they work in an iterative manner throughout the life cycle of the project, rather than realising at the end of the process where they went wrong (Woodhill & Robbins 1998); and (3) participants are able to develop a more nuanced understanding of what success might look like (Roux et al. 2010). Evaluation and reflection activities are further enhanced as learning-focused activities when they are conducted in an engaged, participatory manner (Roux et al. 2010; Woodhill & Robbins 1998). Being able to learn from, and adapt to, complex and ever-changing social-ecological contexts is important for TD research partnerships to remain effective, relevant and responsive (Van Ongevalle et al. 2014).

In TD research, paying particular attention to learning is widely recognised as a critical step, which is often referred to as ‘transdisciplinary learning’ (Roux et al. 2010) or ‘social learning’ (Keen, Brown & Dyball 2005; Reed et al. 2010). Building participatory evaluation and reflection into TD research programmes is one way of making learning processes more explicit (Roux et al. 2010). Cundill, Roux and Parker (2015) point out that more attention needs to be paid to the social processes, such as learning and participatory reflection and evaluation, which support collaborative TD research initiatives. The present study is in part a response to such calls in the literature, and provides insights into such social processes as are experienced in a TD research programme, through its participatory reflection and evaluation activities.

The Durban Research Action Partnership (D’RAP) provides an opportunity to evaluate a TD research programme through a series of actor-oriented evaluation and reflection activities. D’RAP, a joint research partnership between a local university (the University of KwaZulu-Natal) and a local government department (the Environmental Planning and Climate Protection Department of eThekwini Municipality) was established with the intention to bridge the science-policy-practice gap, provide knowledge to assist environmental decision-making and management, and build capacity of both organisations (Cockburn et al. 2016). In local government departments working on environmental management, biodiversity conservation and climate change adaptation, the shortage of human capacity and specialist skills has been recognised across South Africa (Funke & Nienaber 2012; Ivey, Geber & Nänni 2013; Wilhelm-Rechmann & Cowling 2011). Furthermore, the gap between research and action, or science and implementation, is recognised as a barrier to effective environmental management, biodiversity conservation and climate change adaptation in South Africa, and initiatives such as D’RAP are needed to address this hiatus (Knight et al. 2008; Reyers et al. 2010; Sitas et al. 2014).

Cockburn et al. (2016) provided lessons for building a successful TD research partnership. They described the establishment of the D’RAP as a TD research programme for addressing the research-action gap, and shared lessons for building successful research-action partnerships. D’RAP is considered as an example of a TD research programme based on its interdisciplinary approach to addressing real-world problems through collaboration with practitioners and decision-makers, thus bridging the gap amongst disciplines as well as between science and society or practice (Lang et al. 2012).

In the current paper, we present an evaluation of the KwaZulu-Natal Sandstone Sourveld (KZNSS) Research Programme, which falls under the broader D’RAP, to reflect on its effectiveness at an early stage of the partnership. This research programme was the D’RAP’s first programme, and there were initially few existing relationships between researchers and officials, and limited experience of engaging in these types of partnerships. Whilst recognising that it might be too early to assess the research impact on policy development and implementation practices, we share important lessons for establishing continuous learning and reflection processes in TD research programmes. We present a continuous, reflective evaluation based on the participants involved in D’RAP. We specifically aim to evaluate and reflect on (1) research outputs and outcomes; (2) the translation of the research into policy and practice; and (3) participants’ perceptions of the outcomes and processes. We reflect on these participants’ perceptions over the three years of the programme, draw on insights from the literature, and share lessons for evaluation and reflection processes in TD research partnerships.

Method

Case study: The KZNSS research programme

The KZNSS research programme was officially initiated in May 2011 and ran for three years till June 2014. Research projects, conducted almost entirely by postgraduate students, started in January 2012 and the evaluation focused on the period from 2012 onwards. EThekwini Municipality (EM) provided funding of R1 500 000 for the duration of the three-year programme. Co-funding was leveraged and this amounted to an extra contribution of R2 240 000 over the three years. Funding was spent on student bursaries, project running costs, research assistants, and overall project co-ordination.

The programme focused research effort within the KZNSS ecosystem. This ecosystem is found only within the province of KwaZulu-Natal in South Africa, has very high species diversity and endemism, and has been identified provincially as critically endangered (Jewitt 2011). Owing to this conservation status, the municipality is mandated to conserve and manage it effectively (Boon et al. 2016:this issue). The research objectives of the programme were to:

  • improve understanding of biodiversity, ecosystem functioning and ecosystem services
  • improve understanding of past, present and future land use changes
  • assess the effects of climate change
  • develop monitoring protocols in the face of climate change
  • address specific climate change adaptation challenges, including ecosystems-based adaptation
  • assist EM with decision-making for land use planning and policy
  • assist EM in communicating the ecological and socio-economic value of KZNSS
  • build capacity and human capital in the areas listed above
  • develop a learning organisation.
The need for a continuous planning, monitoring and evaluation approach

By the end of 2012, one year into the KZNSS programme, a lack of common understanding of each institution’s research needs, framing, and how to conduct a collaborative TD research process was recognised. This triggered the need to evaluate the process and outcomes of the programme from the end of 2012 onwards (Figure 1). The co-ordination team comprised 6–10 members from both institutions, and was tasked with conducting an evaluation and reflection process. Various activities were conducted (Figure 2, activities 1–11) and followed a generic approach of evaluation, reflection, learning and adaption (Figure 3) (Roux et al. 2010; Van Ongevalle et al. 2014). Each activity was carried out at a different time in the programme period (2012–2014); however, activity 11, the special issue publication from 2016 (Figure 1), is also included as this was a direct output of the programme. For the purpose of clarity, the activities were arranged into three broad evaluation objectives:

  1. Research outputs and outcomes of the programme, including capacity building: assessing outputs (e.g. the number of graduates and research papers published) and outcomes (e.g. increased knowledge of the KZNSS). Traditionally, these include measures of scientific practice recognised by academic institutions (i.e. outputs for research).

  2. Translation of research into societal practice: assessing the suitability and the integration of research into EM environmental policy development and decision-making and management. This objective measures the translation and integration of research into action (i.e. outcomes for practice/implementation).

  3. Participants’ perceptions of, and reflections on, the outcomes and processes of the research programme: evaluating how the participants, from both UKZN and EM, feel about the process, and perceive the overall success of the research programme.

FIGURE 1: Timeline indicating when each evaluation and reflection activity was carried out and which objective each activity aimed to address.

FIGURE 2: Summary of all activities undertaken to evaluate the KZNSS Research Programme arranged according to the three main objectives of the evaluation process.

FIGURE 3: Diagram depicting the continuous process of evaluation, reflection, learning and adaptation, as implemented in the KZNSS programme.

The activities were conducted in a participatory manner with all programme participants in order to embed reflective practices into the programme, and to begin building a learning organisation (Senge 1994), rather than a conventional programme solely focused on generating research outputs. Particular attention was paid to the principles of enabling leadership to create a suitable atmosphere for reflection and learning (Galuska 2014; Uhl-Bien, Marion & McKelvey 2007), which included learning-by-doing, allowing for mistakes, and creating opportunities for participants to question processes as they unfold. This was done by encouraging a flat, rather than a hierarchical structure, and encouraging students and junior academics to participate in meetings and discussions where decisions were made. Discussions took place in a manner that emphasised listening to and respecting diverse viewpoints. Furthermore, team-building activities such as excursions to project sites and social events were seen as opportunities for building relationships of trust and social capital amongst participants (Cheruvelil et al. 2014). The importance of building social capital and collaborative capacity in this case study is discussed elsewhere (Cockburn et al. 2016).

The approach that was followed to assess and report on each evaluation objective is presented below. Further details concerning the specific method of each activity (including time frame and participants) are given in Supplementary Material: Appendix 1.

Assessing research outputs and outcomes

The research outputs were assessed by standard measurements of quantitative scientific productivity (i.e. number of research projects, students trained and graduated, scientific outputs including publications and data). Specifically, a close-out report was compiled which detailed the scientific and collaborative management outcomes, as well as the human capital and social learning outcomes. Documenting the scientific outcomes involved recording the number of students who graduated through the KZNSS under each respective qualification (Honours – a separate 4th year of study following the 3-year Bachelor’s degree, Master’s and PhD). The programme addressed four different themes (biodiversity, ecosystem functioning, land use change and socio-economic changes); therefore each student project was classified accordingly.

Assessing the translation of research into policy and practice

An evaluation of the implementation of research into practice should focus on the extent to which data and knowledge generated by the KZNSS programme influenced policy development, decision-making and management. This is a lengthy process where the influence of research can be felt many years after the programme. Implementation of the KZNSS research programme into practice is still ongoing and only anecdotal evidence is available for assessment. Here, the focus of the evaluation was on the extent to which the research could contribute to policy-development, decision-making and management. This was done by assessing the relevance of research in terms of policy/practice issues raised by eThekwini, alignment of research projects with EM needs, and any anecdotal evidence of translation of research into practice.

The programme first started with a very open approach, and research projects were initiated without a clear focus. There was a tacit understanding that management guidelines would emerge once the research was conducted, following a conventional linear approach to knowledge dissemination. Following the realisation, at the end of 2012, that the research programme was not meeting the needs of the municipality, a research framework was developed (Figure 2, activity 1). This research framework provided a mutual understanding of research needs between the parties and a way to align research with decision-making products (Appendix 1). Formal proposal presentations, presented to the co-ordination committee, were required before new projects were accepted under the programme. Decisions on new projects were made by the co-ordination team (both UKZN and EM representatives). The municipality also provided research questions, in line with the research framework, based on needs and issues of concern to the municipality. These questions were used to guide research topics.

Students provided regular updates in the form of bi-annual (June and December) progress reports and presentations to principal investigators (PIs) and the co-ordination team. After presentations were given by students, discussions were held by the co-ordination team on the extent to which research projects were suitable for translation and integration into EM biodiversity and climate change actions. These group discussions were short (1–2 hours) and took the initial questions posed by the municipality and compared them with student research projects to assess which questions had been addressed.

As part of the students’ progress reports, a section was included where students indicated the alignment of their project with the research objectives of the programme. This section encouraged reflection on how individual students perceived their project and project outputs to be of value and use to the municipality. This component of the research aligns with Cockburn et al. (2016) where an essential action for a successful partnership is to ‘conduct research with implementation in mind’.

Assessing participants’ perceptions

To assess the perceptions of participants in the research programme (including staff/PIs, students from the university and practitioners from the municipality), a mixed methods approach was taken, which included the following activities: an anonymous online questionnaire and follow-up questionnaire (Appendix 2) a year later, individual reflection cards, personal reflection on success and investment over time, and focus group discussions (Appendix 1). The results of the questionnaire and the reflections were collated and analysed using emergent content analysis (Creswell 2009). These activities assessed participants’ perceptions of both the tangible and the intangible outcomes of the programme.

Results

Evaluation of research outputs and capacity building

An objective of the KZNSS research programme was to build capacity and human capital. A total of 29 students were involved in the programme and 26 have graduated through UKZN, as detailed in Table 1. For a three-year programme with relatively limited funding, these figures indicated a high level of capacity building within UKZN, when compared with similar initiatives at the University. Considering the number of projects completed, the resultant publications number was lower than initially hoped for, with only two publications in the ecosystem function theme (McPherson et al. 2016a, 2016b) and one across TD research (Cockburn et al. 2016) (Table 1). However, more than two years after the programme’s funding period finished, 11 publications flowing from this programme have formed part of this journal’s special issue. This highlights the substantial lag phase in producing standard research outputs.

TABLE 1: The number of research projects and scientific publications completed via the 3-year KZNSS research programme (2012–2014) and the number of students funded by eThekwini Municipality (EM) or other funding sources.
Evaluation of translation of research into policy and practice

For the duration of the research programme, several activities were initiated to assess the alignment of the research projects against the research objectives and needs of eThekwini Municipality (Figure 2). This process was important for the municipality to redefine and clarify their research objectives/questions, and for the programme to develop a coherent and reasonable research framework (see Cockburn et al. 2016). The activities were also essential in identifying key gaps in research according to the municipality’s research questions, and realigning the programme where necessary. Major research gaps in governance and climate change issues were identified; however, these were only partly addressed.

At the end of the funding period, whilst most research projects related to EM needs, the data and knowledge generated did not directly translate into practice or policy. This was largely because of issues of format, accessibility and usability of the information, which the municipality identified as barriers to implementation of the research into policy and practice. To address this challenge, the programme has now initiated a process of developing more integrated and implementable knowledge products, such as practitioner guidelines which synthesise the research on the KZNSS ecosystem. Whilst we were unable to assess the impact on formal and informal policy, anecdotal evidence (DR, personal communication) suggests that the partnership highlighted the importance for local government to link to science. The partnership has become a key tool in understanding and managing contemporary urban challenges and provided the platform to facilitate the development and implementation of policies on local land use planning. Three new research programmes are now being funded through this partnership: the city’s Strategic Environmental Assessment, the Community Reforestation Research Programme, and the Global Environmental Change Research Programme.

Interestingly, the research programme yielded some unexpected results which indicate some success in bridging the research-practice-policy gap. For example, through interactions with international researchers and training organised by the programme, EM has now changed their practice around conservation planning and is now using a new software, Zonation (Moilanen 2007), introduced through the programme. This was only made possible through the research links between UKZN and the researchers who developed the Zonation software. With regards to practice, the KZNSS research programme has laid the foundation for interaction between researchers and practitioners around land use planning and policy, through the implementation of a new research programme to develop the eThekwini Municipality Strategic Environmental Assessment. The co-ordination team which was set up for the KZNSS programme has continued working together as new programmes have been added to the overall partnership, and meets on a regular basis. Two research co-ordinators have now been employed to manage the growing partnership.

Evaluation of participants’ perceptions of outcomes and processes

Participants were asked to indicate the level of investment in the programme relative to the level of success they felt was achieved (Figure 4, Appendix 1). This was repeated for each year of the programme from 2012–2014. Some participants only joined the programme in 2013 which explains the increase in respondents over the years. In 2012, there were higher levels of perceived investment relative to success, and this was especially expressed by members of EM (Figure 4). By 2014, the perceived level of success had notably increased from 2012 (more data points in the upper left quadrant of the graph in Figure 4), reflecting a higher level of satisfaction.

FIGURE 4: Participants’ perception of overall success and investment over time. Investment refers to the amount of: time, effort and project funds allocated to the research programme. Success refers to the success of the programme as perceived by the participant. No unit of measure was used to determine investment or success, but rather participants were asked to gauge their response on a scale from low to high.

The evaluation survey helped participants to reflect on their role and participation in the research programme, as indicated by 83% of respondents from 2014 (respondents included 4 EM staff, 9 UKZN staff, and 6 UKZN students). Knowledge generation and training of students were perceived to be the most successful aspects of the partnership. Respondents felt that the greatest factors contributing to the success of the programme were: (1) co-operation, collaboration and commitment of the participants and partner institutions; (2) grant funding and other support; and (3) good communication between the two institutions. The top three challenges that respondents experienced were (1) time constraints; (2) financial and logistical support; and (3) the need to work with different organisational cultures.

From 2013 to 2014, there was a decrease in the percentage of respondents who felt the programme had addressed many of the product-orientated outcomes such as assisting EM with decision making, developing monitoring tools, and communicating the value of the KZNSS (Figure 5a). In 2013, participants were more positive that the programme would address these needs but, by the end of the programme, participants realised that this had not been the case. On the contrary, participants perceived the success of the research programmes more favourably in 2014 with regards to ‘soft’ outcomes and the overall process (Figure 5b). Over 80% of respondents indicated that the programme had helped to increase their ability to work with diverse stakeholders, to develop new long-term work relationships, build new links between organisations, and build trust and mutual understanding among partners (Figure 5b). There was a distinct change in perceptions over time, with a decrease in the perceived success of addressing tangible outcomes and increase in the perceived success of the programme development (e.g., through better collaboration and the building of relationships and trust between partners).

FIGURE 5: Change in participants’ perceptions of (a) outcomes and (b) processes between 2013 and 2014.

In general, the programme appeared to facilitate TD research, and participants appreciated the organic, flexible nature of the programme and the open communication and exchange of ideas. The aspects that participants felt should be improved included an increase in administrative support and a broader research focus, and the leaders in the partnership were able to respond to these concerns and adapt the programme accordingly (within available resources).

Discussion

The process of evaluation and reflection of the KZNSS research programme has been rich in learnings relevant to the programme itself and the broader D’RAP partnership but also to other TD research groups. In particular, we discuss here the role of evaluation and reflection in such partnerships, the challenges of knowledge exchange, and the importance of paying attention to the ongoing research process.

The first key lesson which emerged from the present study was the importance of evaluation and reflection in a TD research partnership. After conducting a wide range of evaluation/reflection activities (Figure 2), the participants gained a richer and deeper understanding of the successes and the challenges of the partnership. Much of that learning would not have been possible without the evaluation and reflection process. For example, without this process, the impact of the programme could not be fully assessed, especially its importance in building the foundations for long-term research partnership between EM and UKZN (Figure 5b). A continuous process of evaluation, reflection, learning and adapting as well as flexibility with participants enabled the programme to readjust and improve as it was happening.

Evaluation and reflection are best done continuously and in a participatory manner to support learning and adaptation (Biggs et al. 2011). Building an atmosphere of reflection, learning and adaptation requires enabling leadership (Roux et al. 2010); it also requires additional resources such as time commitment (Klein 2008). For evaluation and reflection activities to be given appropriate attention, sufficient resources need to be explicitly allocated to this activity, possibly through the appointment of ‘process champions’ who can guide and facilitate such activities (Cockburn et al. 2016; Gray 2008). Lang et al. (2012) and Cockburn et al. (2016) provide several suggestions on guiding principles for TD research which can also help to inform a comprehensive evaluation of TD research partnerships. Through the process of reflective evaluation, we were able to identify the greatest success of the research programme in the less tangible outcomes of building social capital and collaborative capacity, which have laid a firm foundation for future working relationships between research and practice. As this evaluation provides an early assessment of a growing research partnership, we are able to take the learning into the partnership’s future development.

The second key lesson that emerged from the present study was recognising the challenges in exchanging and integrating knowledge. This is not a new finding but it is often under-estimated by stakeholders involved in collaborative research programmes. The KZNSS research programme was only established in 2012 and work on integration is still taking place. As a result, the programme had relatively low levels of integration of research into action compared with the initial expectations of both the researchers and practitioners. A continuous reflective evaluation process, as described in the present study, helped to explicitly consider the translation of research into practice and paved the way for future knowledge integration. Integration of diverse knowledge types, and synthesis of knowledge into useful forms for practitioners and implementers, are widely recognised as significant challenges of TD and implementation-focused research (Pooley, Mendelsohn & Milner-Gulland 2014; Pullin et al. 2016; van Kerkhoff 2014), with some authors even considering integration to be the crux of inter- or transdisciplinarity (Klein 2008; Lang et al. 2012).

The present programme highlighted that the conventional linear approach to knowledge dissemination, initially followed during the first year, is not entirely suitable for research-action partnerships (Lang et al. 2012; van Kerkhoff & Lebel 2006). For scientists to provide recommendations after the research and publication is complete, is not an effective TD practice. TD research seeks ongoing co-generation of knowledge, rather than researchers providing the results to the implementing agent at the end of a research process. One challenge resulting in this traditional ‘trickle-down’ of information is that, in reality, researchers and practitioners are still working in different organisations (i.e. what is often referred to as ‘sitting in silos’) (Pooley et al. 2014), and each comes with its own expectations in terms of conventional knowledge products such as peer-reviewed papers (UKZN) and practitioner-focused management guidelines, policy briefs and handbooks (EM).

Co-generating knowledge through TD research partnership can be a lengthy process (Roux et al. 2010). Appointing consultants to advise on action is an alternative method to obtain results timeously but this is at the expense of a formal peer-review process which validates the results. However, research partnerships such as D’RAP are critical to increase human capital and build long-term datasets to track global change which can hardly be achieved by appointing a consulting company. Such a partnership model between a university and local government should be encouraged as South Africa lacks appropriate human capacity in environmental fields and government departments (Funke & Nienaber 2012; Ivey et al. 2013), particularly in the areas of environmental management, biodiversity conservation and climate change adaptation (Wilhelm-Rechmann & Cowling 2011). TD research programmes should operate over longer time frames (e.g., at least 5–10 years). Such timeframes are typically not aligned with traditional research and practice timeframes (e.g. funding and degree cycles); however, they are necessary to allow sufficient time for building relationships and co-developing integrated knowledge (Klein 2008). Boundary organisations, such as D’RAP, and institutional champions (Franks 2010; Long, Cunningham & Braithwaite 2013) can provide long-term support and stability between funding periods.

The third key lesson which emerged from the present study was that a TD research partnership requires attention to the ongoing process (Klein 2008; Roux et al. 2010). The online questionnaire to assess the successes and challenges of the partnership revealed the importance of ‘soft’, less tangible outcomes (i.e. paying attention to the process of building relationships and not only the research products or outputs) (Figure 5a and 5b). Investing in process rather than product takes much time, effort and commitment in order to build relationships, understanding, and shared decision-making regarding the research programme (Cockburn et al. 2016). Investing in process is often overlooked in research-action partnerships and their evaluation because there is often no obvious product (Fazey et al. 2013).

The importance of these ‘social factors’ in TD research partnerships is widely recognised (Gray 2008; Klein 2008; Sitas et al. 2016), and TD research processes are considered ‘social processes of knowledge production’ (Spaapen, Dijstelbloem & Wamlink 2007). An important lesson from evaluating participants’ perceptions was realising the value of less tangible outcomes. Most participants recognised that we had not yet achieved the more tangible, conventional research outputs of the programme but that we had achieved important networking outcomes such as building social capital and developing collaborative capacity (Figure 5b). The trust building and the laying of a foundation of effective working relationships between the partners was considered a key success factor in the subsequent development of three new research programmes in partnership with EM (Reforestation Research Programme, Global Environmental Change Research Programme, and Strategic Environmental Assessment).

Acknowledgements

The authors acknowledge funding from eThekwini Municipality and from the South African Research Chairs Initiative of the Department of Science and Technology and National Research Foundation of South Africa. The students, principal investigators and other participants in the KwaZulu-Natal Sandstone Sourveld Research Programme are thanked for their participation in this research-action partnership, particularly for the feedback they provided in the process evaluation questionnaires.

Competing interests

The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.

Authors’ contributions

C.T., J.C. and M.R. designed the study and wrote the first draft. J.M., S.M., R.S., D.R., R.B., S.O. and E.D. made conceptual contributions and contributed to subsequent drafts. All authors revised and approved the final copy of the manuscript.

References

Bergmann, M., Brohmann, B., Hoffmann, E., Loibl, M.C., Rehaag, R., Schramm, E. et al., 2005, Quality criteria of transdisciplinary research: A guide for the formative evaluation of research projects, ISOE-Studientexte, no. 13, Frankfurt am Main, Germany.

Biggs, H., Breen, C., Slotow, R. & Freitag, S., 2011, ‘How assessment and reflection relate to more effective learning in adaptive management’, Koedoe 53(2), 1001. http://dx.doi.org/10.4102/koedoe.v53i2.1001

Boon, R., Cockburn, J., Douwes E., Govender, N., Ground, L., Mclean, C. et al., 2016, ‘Managing a threatened ecosystem in an urban biodiversity hotspot: Durban, South Africa’, Bothalia 46(2), a2112. http://dx.doi.org/10.4102/abc.v46i2.2112

Cheruvelil, K.S., Soranno, P.A., Weathers, K.C., Hanson, P.C., Goring, S.J., Filstrup, C.T. et al., 2014, ‘Creating and maintaining high-performing collaborative research teams: The importance of diversity and interpersonal skills’, Frontiers in Ecology and the Environment 12(1), 31–38. http://dx.doi.org/10.1890/130001

Cockburn, J., Rouget, M., Slotow, R., Roberts, D., Boon, R., Douwes, E. et al., 2016, ‘How to build science-action partnerships for local land use planning and management: Lessons from Durban, South Africa’, Ecology and Society 21(1), 28. http://dx.doi.org/10.5751/ES-08109-210128

Cundill, G., Roux, D.J. & Parker, J.N., 2015, ‘Nurturing communities of practice for transdisciplinary research’, Ecology and Society 20(2), 22. http://dx.doi.org/10.5751/ES-07580-200222

Dick, B., 2003, The Snyder evaluation process. A resource file to support the on line program AREOL: Action research and evaluation on line, viewed July 2016 from http://www.aral.com.au/resources/snyder.html

Fazey, I., Evely, A.C., Reed, M.S., Stringer, L.C., Kruijsen, J., White, P.C. et al., 2013, ‘Knowledge exchange: A review and research agenda for environmental management’, Environmental Conservation 40, 19–36. http://dx.doi.org/10.1017/S037689291200029X

Ferreyra, C. & Beard, P., 2007, ‘Participatory evaluation of collaborative and integrated water management: Insights from the field’, Journal of Environmental Planning and Management 50(2), 271–296. http://dx.doi.org/10.1080/09640560601156532

Franks, J., 2010, ‘Boundary organizations for sustainable land management: The example of Dutch environmental co-operatives’, Ecological Economics 70, 283–295. http://dx.doi.org/10.1016/j.ecolecon.2010.08.011

Funke, N. & Nienaber, S., 2012, ‘Promoting uptake and use of conservation science in South Africa by government’, Water SA 38, 105–114. http://dx.doi.org/10.4314/wsa.v38i1.13

Galuska, L., 2014, ‘Enabling leadership: Unleashing creativity, adaptation, and learning in an organization’, Nurse Leader 12(2), 34–38. http://dx.doi.org/10.1016/j.mnl.2014.01.011

Gray, B., 2008, ‘Enhancing transdisciplinary research through collaborative leadership’, American Journal of Preventive Medicine 35, S124–S132. http://dx.doi.org/10.1016/j.amepre.2008.03.037

Hirsch Hadorn, G.H., Hoffmann-Riem, H., Biber-Klemm, S., Grossenbacher-Mansuy, W., Joye, D., Pohl, C. et al., 2008, Handbook of transdisciplinary research, Springer, Dordrecht, Netherlands.

Ivey, P., Geber, H. & Nänni, I., 2013, ‘An innovative South African approach to mentoring novice professionals in biodiversity management’, International Journal of Evidence Based Coaching and Mentoring 11(1), 85–111.

Jahn, T., Bergmann, M. & Keil, F. 2012, ‘Transdisciplinarity: Between mainstreaming and marginalization’, Ecological Economics 79, 1–10. http://dx.doi.org/10.1016/j.ecolecon.2012.04.017

Jewitt, D., 2011, Conservation targets and status for vegetation types in KZN, Unpublished internal report, Biodiversity Conservation Planning Division, Ezemvelo KZN Wildlife, Pietermaritzburg, viewed March 2014, from http://www.bgis.sanbi.org/KZN/KZN_vegetationtypes_conservationtargetstatusreport.pdf

Keen, M., Brown, V.A. & Dyball, R., 2005, Social learning in environmental management: Towards a sustainable future, Earthscan, London.

Klein, J.T., 2008, ‘Evaluation of interdisciplinary and transdisciplinary research’, American Journal of Preventive Medicine 35(2 Suppl.), S116–S23. http://dx.doi.org/10.1016/j.amepre.2008.05.010

Knight, A.T., Cowling, R.M., Rouget, M., Balmford, A., Lombard, A.T. & Campbell, B.M., 2008, ‘Knowing but not doing: Selecting priority conservation areas and the research implementation gap’, Conservation Biology 22(3), 610–617. http://dx.doi.org/10.1111/j.1523-1739.2008.00914.x

Lang, D.J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P. et al., 2012, ‘Transdisciplinary research in sustainability science: Practice, principles, and challenges’, Sustainability Science 7(1), 25–43. http://dx.doi.org/10.1007/s11625-011-0149-x

Long, J.C., Cunningham, F.C. & Braithwaite, J., 2013, ‘Bridges, brokers and boundary spanners in collaborative networks: A systematic review’, BMC Health Services Research 13, 1–13. http://dx.doi.org/10.1186/1472-6963-13-158

McPherson, S.C., Brown, M. & Downs, C.T., 2016b, ‘Crowned eagle nest sites in an urban landscape: Requirements of a large eagle in the Durban Metropolitan Open Space System’, Landscape and Urban Planning 146, 43–50. http://dx.doi.org/10.1016/j.landurbplan.2015.10.004

McPherson, S.C., Brown, M. & Downs, C.T., 2016a, ‘Diet of the crowned eagle (Stephanoaetus coronatus) in an urban landscape: Potential for human-wildlife conflict?’, Urban ecosystems 19(1), 383–396. http://dx.doi.org/10.1007/s11252-015-0500-6

Moilanen, A., 2007, ‘Landscape Zonation, benefit functions and target-based planning. Unifying reserve selection strategies’, Biological Conservation 134, 571–579. http://dx.doi.org/10.1016/j.biocon.2006.09.008

Pooley, S.P., Mendelsohn, J.A. & Milner-Gulland, E.J., 2014, ‘Hunting down the Chimera of multiple disciplinarity in conservation science’, Conservation Biology 28 (1), 22–32. http://dx.doi.org/10.1111/cobi.12183

Pullin, A., Frampton, G., Jongman, R., Kohl, C., Livoreil, B., Lux, A. et al., 2016, ‘Selecting appropriate methods of knowledge synthesis to inform biodiversity policy’, Biodiversity and Conservation 25(7), 1285–1300. http://dx.doi.org/10.1007/s10531-016-1131-9

Reed, M.S., Evely, A.C., Cundill, G., Fazey, I., Glass, J., Laing, A. et al., 2010, ‘What is social learning?’, Ecology and Society 15(4), r1.

Reyers, B., Roux, D.J., Cowling, R.M., Ginsburg, A.E., Nel, J.L. & Farrell, P.O., 2010, ‘Conservation planning as a transdisciplinary process’, Conservation Biology 24, 957–965. http://dx.doi.org/10.1111/j.1523-1739.2010.01497.x

Rossi, P.H., Lipsey, M.W. & Freeman, H.E., 2003, Evaluation: A systematic approach, Sage, California.

Roux, D.J., Stirzaker, R.J., Breen, C.M., Lefroy, E.C. & Cresswell, H.P., 2010, ‘Framework for participative reflection on the accomplishment of transdisciplinary research programs’, Environmental Science and Policy 13(8), 733–741. http://dx.doi.org/10.1016/j.envsci.2010.08.002

Senge, P.M., 1994, The fifth discipline fieldbook: Strategies and tools for building a learning organization, Currency Doubleday, New York.

Sitas, N., Prozesky, H., Esler, K. & Reyers, B., 2014, ‘Exploring the gap between ecosystem service research and management in development planning’, Sustainability 6, 3802–3824. http://dx.doi.org/10.3390/su6063802

Sitas, N., Reyers, B., Cundill, G., Prozesky, H.E., Nel, J.L. & Esler, K.J., 2016, ‘Fostering collaboration for knowledge and action in disaster management in South Africa’, Current Opinion in Environmental Sustainability 19, 94–102. http://dx.doi.org/10.1016/j.cosust.2015.12.007

Spaapen, J., Dijstelbloem, H. & Wamlink, F., 2007, Evaluating research in context: A method for assessment, 2nd edn., Consultative Committee of Sector Councils for Research and Development, The Hague, Netherlands.

Uhl-Bien, M., Marion, R. & McKelvey, B., 2007, ‘Complexity leadership theory: Shifting leadership from the industrial age to the knowledge era’, Leadership Quarterly 18(4), 298–318. http://dx.doi.org/10.1016/j.leaqua.2007.04.002

Van Kerkhoff, L., 2014, ‘Developing integrative research for sustainability science through a complexity principles-based approach’, Sustainability Science 9(2), 143–155. http://dx.doi.org/10.1007/s11625-013-0203-y

Van Kerkhoff, L. & Lebel, L., 2006, ‘Linking knowledge and action for sustainable development’, Annual Review of Environment and Resources 31(1), 445–477. http://dx.doi.org/10.1146/annurev.energy.31.102405.170850

Van Ongevalle, J., Huyse, H. & Van Petegem, P., 2014, ‘Dealing with complexity through actor-focused planning, monitoring and evaluation (PME)’, Evaluation 20, 447–466. http://dx.doi.org/10.1177/1356389014551487

Wilhelm-Rechmann, A. & Cowling, R.M. 2011, ‘Framing biodiversity conservation for decision makers: Insights from four South African municipalities’, Conservation Letters 4(1), 73–80. http://dx.doi.org/10.1111/j.1755-263X.2010.00149.x

Woodhill, J. & Robins, L., 1998, Participatory evaluation for landcare and catchment groups: A guide for facilitators, Greening Australia, Canberra.

Appendix 1:

Detailed method for each activity in the evaluation process
Activity 1: Developing the research framework

Aim: The purpose of the first activity was to develop a mutual understanding between parties to structure research and ensure the relevance of research to the eThekwini Municipality (EM). The conceptual research framework and decision-making products were developed as a means to direct future research activities (see Cockburn et al. 2016; Figure A4.1.)

Method: A first draft was developed with the University of KwaZulu-Natal (UKZN) principal investigators during a 1-day workshop in November 2012. This was subsequently revised by the co-ordination team which included members from both the municipality and university. The research framework was improved as needed for the duration of the research programme (e.g. a social-economic component was added later).

Activity 2: Bi-annual project report

Aim: The project report was to assess progress and alignment of projects with the programme research objectives from the perspective of students and principal investigators (PIs).

Method: Students, together with their respective PIs, completed a report per student project twice a year for the duration of their research project. Report templates were provided and students commented on the relevance of their project to the needs of EM. The following questions, posed by EM, were included in the report for students to respond to:

  1. Does this project address specific climate change challenges?

  2. Does this project address climate change adaptation issues, especially ecosystems-based adaptation?

  3. Does this project have synergy with other projects in the programme and, if so, has this lead to inter-project collaboration?

  4. Will this project have practical management benefits for eThekwini Municipality? What are they?

  5. Will this project help eThekwini Municipality officials in communicating the benefits of biodiversity and the impacts of climate change?

  6. Will this project result in a better understanding of ecosystem services?

  7. Does this project assist eThekwini Municipality with conservation planning?

This activity served as a personal project reflection for students and PIs to adjust and align their projects to the needs of EM. Through this continuous reporting process, any misalignment of projects was identified and addressed accordingly. This process also gave the opportunity for students to consider the impact of their research on management and practice.

Activities 3 and 7: Matching projects to research framework and gap analysis

Aim: To determine if the research framework is adequate and if the current research projects address the needs of the municipality from the perspective of the co-ordination committee (both UKZN and EM staff).

Method: The co-ordination committee met in June 2013 to compare the research framework against the current student projects at that time and against the needs of the municipality (Activity 3). For this discussion, EM provided a list of information they required and questions they had (Table A1). These questions where then compared with the student projects being conducted. By this method, the co-ordination committee was able to identify research gaps. The gap analysis (comparison of research projects with EM questions) was repeated a year later in September 2014 as new research projects had started in 2014 (Activity 7).

TABLE 1-A1: List of questions provided by EM.
Activities 4 and 8: Online anonymous questionnaire and follow-up questionnaire

Aim: To understand participants’ perceptions of the programme, collect baseline data and identify issues. The questionnaire was initiated owing to discontent in the programme.

Method: An online anonymous questionnaire was designed using the SurveyMonkey software. This was distributed in 2013 to everyone involved in the KZNSS research programme from both EM and UKZN (Activity 4). The same questionnaire was repeated a year later in 2014 as a follow-up (Activity 8). For the first and second follow-up questionnaire, there were 29 and 19 respondents respectively, with 9 respondents who took part in both questionnaires. The respondents included students, UKZN and EM staff. See the detailed questionnaire in supplementary material Appendix 2. Noteworthy results from the two online questionnaires are presented in Figure 5.

Activity 5: Reflection questions

Aim: To help to identify strengths and weaknesses of the partnership. To triangulate answers with the questionnaire.

Method: This took the form of a mini reflection where participants were asked a few short questions. This mini reflection was conducted in October 2013 at a group meeting where students had been presenting their research to EM and UKZN staff, therefore students, PIs and EM staff all participated in the reflection. Participants had to respond to the following four questions (Questions 1 and 2 contribute to evaluation data and 3 and 4 are personal reflections):

  1. One aspect of the research partnership you would like to change

  2. One aspect of the research partnership you would NOT like to change

  3. What can YOU do to make the programme better?

  4. What can WE do to make the programme better?

Activity 6: Perceived success v. investment

Aim: To evaluate the perceived success v. investment in each year of the partnership. This measure served as a rapid check on the change-over time.

Method: This was another mini reflection conducted at a full team meeting where students had presented their research to EM and UKZN staff. Each participant was given a blank graph (Figure A1) and had to position the relationship between investment (i.e. time, effort and project funds spent by the respondent or his/her institution) and the success of the programme (as perceived by the respondent/institution). This was done in 2014, so participants reflected back on 3 years of the programme (2012, 2013 and 2014). If a respondent indicated that a particular year fell below the diagonal dashed line, they perceived their investment into the programme to be greater that the success achieved. However, if the year in question was depicted above the diagonal line, the respondent perceived the success to be greater than the investment. The x and y axes provide a relative scale of increasing proportion as the level of success or investment. Results from this activity are presented in Figure 4.

FIGURE 1-A1: Template graph for respondents to demonstrate their level of investment in the KZNSS programme against their perceived success of the programme.

Activity 9: Close-out workshop

Aim: The purpose of this workshop was to reflect on successful and unsuccessful aspects of the partnership and explore future goals through an in-depth reflection process. Learning through this process was also incorporated in the planning for phase two.

Method: A 2-day workshop was held to reflect on the research that had been conducted, considering how it could inform practice, where the major gaps were, and how practice could inform research. Various activities were organised to facilitate the reflection and discussion process.

The reflection process was divided into three parts. The first was a paired verbal reflection with another workshop participant. The second was a 20-minute personal reflection, and the last was a feedback session to share thoughts and ideas. The following questions were suggested for each reflection:

  1. What inspires me and energises me in the work I do?

  2. What is MY role in this research-action partnership?

  3. What is the role of students/researchers/practitioners in research-action partnerships? (Choose one.)

  4. What is the role of this research-action partnership in society?

  5. Aspects of the partnership you would like to change.

  6. Aspects of the research partnership you would NOT like to change.

  7. What can YOU do to make the programme better?

  8. What can WE do to make the programme better?

Other discussions were held throughout the workshop, exploring methods of knowledge integration and integration of science into practice. The benefit of these discussions was having input from academics (UKZN) and practitioners (EM).

Activity 10: Close-out report

Aim: To summarise the research activities and outcomes of the 3-year programme period.

Method: The current document was compiled at the end of the 3-year research programme and detailed all activities, outcomes, research and processes carried out. The report was compiled internally by members of the UKZN co-ordination committee with input from EM staff.

The close-out report can be accessed online: http://www.durban.gov.za/City_Services/development_planning_management/environmental_planning_climate_protection/Publications/Documents/KZNSS_Close_out_Report2011_2014.pdf

Activity 11: Special issue publication

Aim: To publish peer-reviewed articles of new knowledge generated through the research programme.

Method: Researchers were invited to voluntarily submit manuscripts to be included in a special issue of a peer-reviewed journal. A special issue compilation was decided on as all manuscripts touched on various aspects of the KZNSS. Where possible, authorship of papers included members from both institutions, adding to the aims of co-generation of knowledge. Guest editors also represented both EM and UKZN. The special issue includes 11 manuscripts adding to the knowledge of the KZNSS across various disciplines.

Appendix 2:

Online questionnaire
Questions
  1. Describe your role in the partnership

  2. Please choose which of these best describes your role?

    - Student

    - Principal investigator

    - Associated researcher

    - Manager for eThekwini Municipality

    - Other (specify)

  3. When did you join?

  4. How would you rate the partnership overall?

    - Extremely satisfactory – Satisfactory – Indifferent – Unsatisfactory – Extremely unsatisfactory

  5. Which aspects of the partnership would you say have been successful? [Y/N, multiple choice]

    - Increased funding

    - Knowledge generation

    - Linking with outside institutions

    - Transfer of knowledge

    - Training of students

    - Innovative solutions

    - Other

  6. What factors have contributed most to the success of the partnership? Rank from 1 (high) to 3 (low).

    - Co-operation, collaboration, commitment of team and partnering institutions

    - Common vision, mission, goals

    - Champions for the initiative

    - Good communication

    - Grant funding and other support

    - Increasing the number of participants in the partnership

    - Mutual respect for the strengths of others

    - Adaptive management strategies of the partnership

    - Other (specify and rank)

  7. What do you feel have been your greatest challenges? Rank from 1 (high) to 3 (low).

    - Obtaining appropriate attention from the other partner (EM or UKZN)

    - Time constraints

    - Changes due to reorganisation

    - Finding suitable participants (e.g. students, PIs)

    - Financial and logistical support

    - Need to work with different organisational cultures

    - Short-term partnership

    - Other (specify)

  8. How would you address the challenges identified?

The section below addresses the ‘perceived’ outcomes of the partnership.
  • 9. Do you feel you have acquired a better understanding of:
      Yes Somewhat No
    KZNSS ecosystem      
    Land use planning      
    Climate change (CC) adaptation and mitigation      
    Management and policy      
    Common goal and partnership      
    Other (specify)      
  • 10. Do you feel that the KZNSS partnership (in general) has helped to:
      Yes Somewhat No Can’t say
    Increase understanding of biodiversity of KZNSS        
    Increase understanding of ecosystem functioning and ecosystem services        
    Better understanding of past, present and future land use change        
    Assist EM with decision-making for land use planning        
    Assist EM with decision-making for management        
    Communicate the ecological and socio-economic value of KZN SS        
    Develop monitoring protocols in the face of climate change        
    Address specific climate change challenges        
    Address climate change adaptation issues, especially ecosystems-based adaptation        
    Increase human capital in the areas listed above        
  • 11. Do you feel that the partnership has helped to:
      Yes Somewhat No Can’t say
    Bridge the science-policy gap        
    Increase confidence of individuals to address management of threatened ecosystems        
    Increase ability to work with multi-stakeholders        
    Develop new long-term work relationships amongst individuals        
    Develop new links amongst organisations        
    Build trust and mutual understanding amongst partners        
    Create knowledge that is solution-oriented, socially robust and transferable to both the scientific and EM practice        
    Provide a platform for innovation and applied research        
  • 12. List any unintended outcomes of the partnership (positive or negative).
  • 13. Have you been involved in similar partnerships before? (Note: moved up in the questionnaire.)
  • 14. What makes working in this partnership different than other academic research projects?
  • 15. Would you recommend this partnership to other academics, students or EM staff?
  • 16. Would you like to continue being involved in this partnership?
  • 17. How do you feel about the logistics and support provided with regards to:
      Too much Fine Too little
    Communication      
    Administration      
    Reporting      
    Finance      
    Meeting      
  • 18. What do you think is missing from this partnership?
  • 19. What do you think is missing from this evaluation?
  • 20. Any other comment(s)?

 

Crossref Citations

1. Revealing mechanisms in a transdisciplinary Community Reforestation Research Programme
Michael R. Ward, Eureta Rosenberg
African Evaluation Journal  vol: 8  issue: 1  year: 2020  
doi: 10.4102/aej.v8i1.467