framework for Marine Parks Program

(1)

Monitoring, Evaluation and Reporting

framework for Marine Parks Program

DEWNR Technical note 2017/06

(2)

Monitoring, Evaluation and Reporting framework for Marine Parks Program

Glen Scholz, Patricia von Baumgarten, Hugh Wilson, Alison Wright and Simon Bryars

Department of Environment, Water and Natural Resources

DEWNR Technical note 2017/06

(3)

Department of Environment, Water and Natural Resources GPO Box 1047, Adelaide SA 5001

Telephone National (08) 8463 6946 International +61 8 8463 6946 Fax National (08) 8463 6999

International +61 8 8463 6999 Website www.environment.sa.gov.au

Disclaimer

The Department of Environment, Water and Natural Resources and its employees do not warrant or make any representation regarding the use, or results of the use, of the information contained herein as regards to its correctness, accuracy, reliability, currency or otherwise. The Department of Environment, Water and Natural Resources and its employees expressly disclaims all liability or responsibility to any person using the information or advice. Information contained in this document is correct at the time of writing.

This work is licensed under the Creative Commons Attribution 4.0 International License.

To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

ISBN 978-1-925510-41-6

Preferred way to cite this publication

Scholz G, von Baumgarten P, Wilson H, Wright A and Bryars S, 2017, Monitoring, Evaluation and Reporting framework for Marine Parks Program, DEWNR Technical note 2017/06, Government of South Australia, through the Department of Environment, Water and Natural Resources, Adelaide

Download this document at: https://data.environment.sa.gov.au/Coast-and-Marine/Coast-Marine- Management/Pages/home.aspx

(4)

1 Purpose

1.1 Purpose

The Government of South Australia has committed to developing the South Australian Representative System of Marine Protected Areas. The Department of Environment Water and Natural Resources (DEWNR) lead the development of the marine parks network to encompass the major ecosystems and habitat types within and between each of the state’s eight bioregions. To provide the legislative base to protect South Australia’s marine environment, the Marine Parks Act 2007 (the Act) as proclaimed. Under the Act, the Minister for Sustainability, Environment and Conservation (the Minister) is required to review management plans within a 10 year period. The monitoring, evaluation and reporting program (MER Program) will provide critical environmental, economic and social information to inform this review.

This document describes the framework for monitoring, evaluating and reporting (MER) on the effectiveness of the South Australian Marine Parks network in delivering on the objects of the Act in accordance to Strategy 10 of Marine Parks’ Management Plans. This framework provides the direction and outlines the steps and components required to develop the MER Program. The MER Program is intended to provide for public accountability and continuous improvement of the Marine Parks network as part of an adaptive management cycle (Figure 1).

The objectives of the MER Program include:

- Providing evaluation based evidence on how effective the Marine Park Management Plans are in delivering a Comprehensive, Adequate and Representative network as per the requirement of the primary object of the Act

- Providing data and information to support effective implementation, operation and improvement of the Marine Parks Program as a whole, including planning

- Undertaking evaluations to support continuous improvement of the delivery of Marine Parks - Reporting of the status of marine parks performance, and performance of the Marine Parks

Program as a whole

- Promoting accountability as it provides the necessary information on management effectiveness that allows for an assessment of whether results are being achieved that are commensurate with the efforts and resources being expended, and in line with the objects determined by legislation.

It is envisaged that results of the MER Program will inform the review of the Marine Park Management Plans, which will be undertaken at least every 10 years in accordance with the requirements of the Act.

This MER framework establishes the components that underpin the MER program, and enables it to provide for the monitoring, evaluation and reporting activities required for an evidence-based review process. The framework also provides for shorter term adaptive management of the activities and components contributing to implementation of management plans.

(7)

Figure 1. Marine Parks Program – Adaptive Management Cycle

This adaptive management cycle highlights how the Marine Parks Management Plan strategies are implemented by the four management sub-programs, and through a monitoring, evaluation and reporting process inform the review of the

management plans.

This MER framework covers 1) core principles, context and definitions and 2) an outline of the key components, including context, objectives, key evaluation questions, monitoring, evaluation and reporting requirements. It draws on best practice monitoring, evaluation and reporting approaches, including: Australian Government Natural Resources Management (NRM) MERI framework (Australian Government, 2009); Monitoring and Evaluation for Adaptive Management: Issues Paper (National Water Commission, 2013); Science guidelines to support water allocation plans – ecology, hydrology and hydrogeology Part 6: Monitoring, evaluation, reporting and improvement, (Department of Environment, Water and Natural Resources, 2014) and Developing Monitoring and Evaluation Frameworks (Markiewicz and Patrick, 2016).

(8)

2 Background, context and principles

2.1 Monitoring, Evaluation and Reporting Program for the South Australian Marine Parks network

A MER Program for the marine parks network was established in 2012 as provided for in Strategy 10 of Marine Parks’ Management Plans as follows:

Develop and implement a monitoring, evaluation and reporting (MER) program that measures the effectiveness of this marine park management plan and its contribution to South Australia’s marine parks network (2011 baseline), and:

 is designed to measure the effectiveness of the management plan in delivering the predicted outcomes to inform adaptive management

 includes linkages to relevant state, national and international monitoring, evaluation and reporting frameworks

 sets out targets and indicators linked to strategies and outcomes for environmental and management elements

 monitors the delivery of education, research and governance mechanisms

 assesses the effectiveness of compliance activities.

2.2 Definitions

The following definitions, being used for MER Program purposes, have been adopted from DEWNR 2014:

 Monitoring: To watch – routine collection of quantitative or qualitative information for the purposes of reporting and/or evaluation.

 Evaluation¹: A structured process of inquiry to discover the worth or relevance of plans, policies, activities, assumptions, decisions or other factors impacting the achievement of planned outcomes. In the marine parks context, evaluation has been defined broadly as “the judgement or assessment of achievement against the objects identified in the Marine Parks Act 2007”.

 Reporting: Routine communication of monitoring and evaluation outcomes to stakeholders for the purposes of accountability and informed decision making.

 Improvement: “Closing the loop” to ensure that findings of monitoring, evaluation and reporting are considered in decision making with respect to planning or implementation.

1In this document the terms evaluation and assessment are used interchangeably.

(9)

2.3 Principles

The Marine Park MER program adopts the following guiding principles for monitoring, evaluation and reporting activities:

2.3.1 Planning is essential for MER to support adaptive management, good governance and knowledge management

Planning for monitoring, evaluation and reporting increases the likelihood that these processes deliver value to programs and activities. Planning is essential to ensure that:

 the scope of monitoring and evaluation activities target the most relevant issues

 evaluation and reporting is timed to influence key decisions affecting program direction and performance

 evaluations have timely access to the right data, information and knowledge to address key evaluation questions

 the right stakeholders participate in monitoring, evaluation and reporting processes to maximise opportunities for learning and for program improvement.

Planning for monitoring, evaluation and reporting typically involves the development of a schedule outlining when specific evaluation and reporting activities occur, by whom, by when, and in response to what needs.

2.3.2 Planning for MER starts with a clear understanding of how a program is anticipated to achieve planned outcomes

Monitoring, evaluation and reporting activities deliver most value when based on a clear understanding of the outcomes to be achieved and the mechanisms by which planned activities will, in time, contribute to these outcomes. Typically, program logic is used as a methodology for developing and explaining a programs cause and effect relationships showing a program’s pathways from a program’s allocated resources (inputs), activities, outputs, and shorter and longer term outcomes. It also highlights assumptions and external factors influencing the program:

 Outcomes in program logic are generally categorised according to timeframes over which they are expected to be observed (current activities, immediate outcomes, intermediate outcomes, long-term outcomes)

 Assumptions are statements or hypotheses that are believed to be true concerning how and why a program is believed to work in a certain context.

 External factors are factors that are generally outside of the program’s control that interact with and influence the program in a positive or negative manner.

2.3.3 Planning for monitoring and evaluations should be informed by established key evaluation questions (KEQs) before identifying indicators

Monitoring is the routine collection of quantitative and qualitative information. It must be clear how data derived from monitoring will be applied to meet the information needs of evaluation and reporting processes. Key evaluation questions (KEQs) provide focus for data gathering efforts thereby increasing the likelihood that the right data are collected so evaluation and reporting can lead to improvement and good governance.

(10)

To determine the effectiveness of an implemented program KEQs seek to address the following criteria:

 Effectiveness – relates to the success at achieving stated objectives.

 Impact – relates to the intended and unintended, positive and negative outcomes of a program or plan.

 Efficiency – relates to the extent to which resources committed to the development and implementation of a program or plan have contributed to outcomes.

 Appropriateness – relates to the extent to which the right objectives, processes and provisions have been established and implemented.

 Sustainability – relates to the extent to which benefits of the program are ongoing.

2.3.4 Considering multiple lines of qualitative and quantitative evidence as part of evaluation processes

Evaluation processes may consider multiple evaluation questions that often cannot be addressed by a single source of evidence or analytical design. Drawing on multiple lines and levels of evidence enables determination of both trends in resource condition and the extent to which programs have contributed to resource condition outcomes.

2.3.5 Facilitate improvement through participatory approaches to monitoring, evaluation and reporting

The MER framework conceptualises MER as a continuous cycle of participation and communication rather than a single evaluation event. It proposes that active participation of planners and decision makers in evaluation facilitates learning and increases the likelihood that key findings are considered by program planning and implementation decisions.

2.3.6 Monitoring for evaluation - not monitoring for monitoring

This MER framework adopts a primary focus on “what knowledge is required” rather than what can be readily measured. As such, monitoring serves the requirements of evaluation and reporting, and is not an entity on its own.

(11)

3 Components of the MER framework for Marine Parks Program

The Marine Parks Program has been developed around four management sub-programs with

associated strategies these are Protection (policy, planning, permitting and governance), Stewardship (education and engagement), Performance (monitoring evaluation and reporting) and Compliance (investigation and enforcement). The MER framework is intended to address the monitoring, evaluation and reporting needs of the Marine Parks Program as a whole in an integrated fashion.

The MER framework is comprised of eight components, as described in Table 1, which outline the total scope of MER for Marine Parks. These components are intended to be developed sequentially, with later components informed by the products of preceding component. The MER Plan (Step 5 as distinguished from the framework) is informed by the outcomes of the earlier foundational stages of the framework and focuses on the strategic definition of the programs evaluative approach giving direction to the programs monitoring requirements. The MER Plan then refers to the operational plans to guide and implement the activities. Some components, such as the Program Logic and

Implementation Schedule, are living documents that will be updated periodically to inform and respond to the adaptive management requirements of the program.

Table 1 also outlines a number of internal and external information products, such as reports, lists (or registers) and plans, to be produced for each component of the framework. These products have different formats and communication mechanisms depending on their purpose and audience.

Table 1. Components of the MER framework for Marine Parks Program MER framework

components

Purpose Content Format and Products

3.1 Scope and

objectives Articulate scope of MER program by defining purpose, requirements and parameters and define objectives for MER in context of the Marine Parks Program (MPP)

Define key evaluation questions (KEQ) to be addressed by MER.

MER program objectives and

audiences (policy plans and programs) Key evaluation questions

Scope of MER activities, stakeholders and partners

Marine Parks MER

Framework (this document) Key evaluation questions fact sheet

3.2 Program Theory and Program Logic (including key assumptions and external factors)²

Articulate rationale for how the Marine Parks Program is anticipated to achieve the objectives of the Act identifying the causal links and relationships that exist between what the program does and the results it is

Program theory for the MPP highlighting key cause and effect relationships

Program logic presenting a timeline of activities, outputs and anticipated short term, intermediate and long term outcomes for the MPP

Document presenting the program theory and program logic Information product presenting a conceptual model of the program logic

2 Refer to section 3.2 for difference between Program Theory and Program Logic as adopted by this framework.

(12)

expected to achieve, from project activities to policy outcomes.

List of key assumptions underpinning outcomes presented in program logic.

Identify those that are dependent on factors internal to the MPP versus those not controlled by the MPP.

A register of assumptions (Publicly available web page and downloadable documents)

3.3 Baseline Reports

Benchmarking the current knowledge base of the Marine parks system in relation to the themes identified by the KEQs

• Values

• Pressures

• Drivers (external)

• Management planning

• Predictions and indicators of change

Series of reports based on each of the 19 Marine Parks plus a statewide

consolidation report 3.4 Key evaluation

questions and subordinate evaluation questions (Generation of evaluation questions)

KEQs set direction and provide focus for monitoring and evaluation activities.

Frame and prioritise subordinate evaluation questions required to address KEQs and meet MER objectives from program logic.

Prioritise evaluation questions according to their importance in delivering MER objectives.

Comprehensive list of evaluation questions addressing assumptions derived from the program logic and informed by the baseline reports Assessment of risks to MER and MPP objectives caused by not addressing each evaluation question, with risks evaluated as low, medium and high

Register of evaluation questions with links to key assumptions of the program logic

A section of the MER Plan document

3.5 Monitoring Evaluation Reporting (MER) Plan

Describe the monitoring, evaluation and reporting aspects of the program that highlights the scientific rationale for the monitoring, evaluation and reporting program (with outcomes from Steps 1 to 4); including supporting information and evidence from the baseline reports; and refers to the strategic links to the operational plans.

Agree on MER priorities for investment with partners and stakeholders based on the theory of change from baseline reports

Plan high level monitoring, evaluation and reporting activities based on agreed priorities

Establish knowledge management procedures for managing information

Incorporating a summary of the outcomes of Steps 1 to 4

Provisional planning and resourcing of monitoring, evaluation and reporting activities for all subordinate evaluation questions having medium and high priority (risk)

Engagement with partners and stakeholders to agree final MER investment priorities on the basis of costs and benefits

A final MER plan outlining:

1) Monitoring, evaluation and reporting activities for agreed priority subordinate evaluation questions 2) The synthesis of evaluation and reporting products to address the key evaluation questions.

High level schedule of key program deliverables

Knowledge management approach accountabilities, protocols, resources in alignment with the DEWNR Information Management Framework IMF and Managing Environmental Knowledge (MEK)

Provisional (internal) draft plan for partner and stakeholder engagement Final (public) MER plan addressing agreed MER priorities

• Marine Park data management standards

• Marine Parks knowledge catalogue

• Marine Park MEK completed forms

(13)

3.6 Operational Planning

Outline the key operational strategies and activities required by the Marine Parks Program four sub-programs (Protection, Stewardship, Performance and

Compliance) to deliver the monitoring and evaluation activities identified in the MER Plan.

Different type of documents that describe the operational strategies and activities guided by the MER Plan Marine Parks Operational Plans Describes the monitoring program, the annual monitoring schedule, the methods and protocols used and the resources required

Research Strategy

Describes the monitoring and research objectives for informing the Marine Parks MER Program

Annual Marine Parks Project Plans (Protection, Stewardship, Performance and Compliance)

Annual Ecological

Monitoring Status Reports Annual Compliance Reports Annual Socio-Economic Monitoring Status Reports

‘Forging the Links’ Research prospectus 2013-2015 Marine Parks Program Research Strategy 2015- onward

3.7 Communication Plan

The Marine Parks

Communications Strategy is an overarching document that provides direction to the Marine Parks Performance Communications Plan The Marine Parks Program Communications Plan outlines a framework for effective and timely communication of the Marine Parks MER activities, results and findings as the program evolves

Outlines key audiences, what they want to know, and when and how the information will be provided and media delivery platforms for:

• Communication of program reporting outputs and outcomes

• Classification of products according to whether they are for public or internal use.

• Promotion of research strategy Education and information

• Open access data sharing

Marine Parks Performance Communications Plan (Internal)

Products via:

• Marine Parks web page

• Enviro Data SA website

3.8 Implementation Schedule

Outlines MER program delivery schedule (work plan), milestones, specific assigned responsibilities, and products required to achieve agreed outputs of the final MER plan

Presented as a Gantt chart

Outlines ‘The When’

Schedule outlining:

 Timing/frequency of monitoring and evaluation activities; and

 Timing/frequency of reporting/communications Identification of responsible parties for monitoring, evaluation and reporting activities

Plan for an evaluation of this MER framework and the MER program it directs

Comprehensive schedule of all activities and releases of:

• Internal products (DEWNR Intranet document)

• External products (published on website)

• RIAS Reports (Release 2015)

Baseline Reports: 19 Marine Parks and 1 statewide (release 2016)

Annual MP Monitoring Operational Reports (bioregional ecological and statewide social and economic) (first release June 2017)

(14)

3.1 Scope and objectives

The Act requires the Minister to review marine park management plans at least once in every 10 years.

The Minister may also propose the amendment of a management plan at any time. Therefore, the MER program aims to monitor, evaluate and report the effectiveness of the marine park management plans so its results can be used to:

1. inform the 10 yearly statutory review of management plans

2. inform any amendments to management plans required outside of the statutory review cycle.

The MER Program also aims to contribute to the public accountability of marine park management plans through regular reporting of performance to the South Australian community.

A key objective of the MER program is to evaluate how effective the Marine Parks Management Plans are on delivering the objects of the Act. To that extent, evaluation of the effectiveness of the

management plans in protecting and conserving marine biological diversity and marine habitats through a comprehensive, adequate and representative (CAR) system is the primary and most

fundamental objective of the MER Program. However, the MER Program also includes an evaluation of the effectiveness of Marine Parks’ Management Plans ’to assist in’ the achievement of other objectives, as highlighted below, in accordance with the Marine Parks Act:

(a) to protect and conserve marine biological diversity and marine habitats by declaring and providing for the management of a comprehensive, adequate and representative system of marine parks; and

(b) to assist in—

(i) the maintenance of ecological processes in the marine environment; and

(ii) the adaptation to the impacts of climate change in the marine environment; and (iii) protecting and conserving features of natural or cultural heritage significance;

and

(iv) allowing ecologically sustainable development and use of marine environments;

and

(v) providing opportunities for public appreciation, education, understanding and enjoyment of marine environments.

Marine Parks’ Management Plans, as key statutory documents, describe 15 strategies to be

implemented for the purpose of delivering on the legal objects described above. These strategies are:

1. Manage activities and uses in the marine park in accordance with zoning and special purpose area provisions.

2. Actively influence activities and uses within and adjacent to the marine park to help mitigate threats to marine biodiversity and marine habitats.

(15)

3. Consider additional protections and/or temporary restrictions where necessary in circumstances of urgency:

(a) to protect a listed species of plant or animal, or threatened ecological community; or (b) to protect a feature of natural or cultural heritage significance;

or

(c) to protect public safety.

4. Introduce a permitting system to provide for the following activities (where not otherwise authorised):

 scientific research in a sanctuary or restricted access zone

 tourism operations in a sanctuary zone

 competitions and organised events in a sanctuary zone

 commercial film-making (including sound recording and photography) in a sanctuary zone

 installation of vessel moorings in a sanctuary zone.

5. Provide for public appreciation, understanding and enjoyment of the marine park.

6. Create and promote opportunities for sustainable nature-based tourism in the marine park.

7. Provide education to support the implementation of the marine park.

8. Seek to involve local communities and stakeholders in the day-to-day management and monitoring of the marine park.

9. Work cooperatively with Aboriginal communities to conserve country, plants, animals and culture.

More specific to the Performance sub-program Strategies 10–14 below, stated in each Marine Park Management Plan, provide for the MER program as follows:

10. Develop and implement a Monitoring, Evaluation and Reporting (MER) Program that measures the effectiveness of this Marine Park Management Plan and its contribution to South Australia’s Marine Parks network (2011 baseline), and that:

 includes linkages to relevant state, national and international monitoring, evaluation and reporting frameworks

 sets out targets and indicators linked to strategies and outcomes for monitoring, which include ecological, socio-economic, environmental and management elements

 monitors the delivery of education, research and governance mechanisms

 assesses the effectiveness of compliance activities.

In addition it will:

11. Foster partnerships to support the implementation of the MER Program incorporating opportunities for community and stakeholder involvement

12. Ensure outcomes of the MER Program and research outcomes are made publicly available and inform decision making and periodic review of the management plans

(16)

13. Conduct priority research and foster research partnerships to assess the integrity of knowledge frameworks that underpin the predicted outcomes

14. Encourage Aboriginal people, local communities and stakeholders to preserve traditional and historic knowledge and, where appropriate, share this knowledge with others.

In light of the strategies listed above, the scope of the MER program includes all aspects covered by these strategies. The MER Program adopts the principle that an evaluation of the effectiveness of the strategies of the Management Plans is a key component of the evaluation of both the Management Plans and the Program as a whole.

15. Develop and implement a compliance strategy for the marine park that:

 is cost-efficient

 is focussed on sanctuary zones and other conservation priorities

 complements existing compliance efforts

 maximises voluntary compliance

 includes measures to address serious or repeat non-compliance.

The Management Plan review (10 yrs) should focus on the extent to which the Marine Parks Program and the Marine Parks management plans have effectively achieved the objects of the Act and managed risks to resources, risks to community values and risks to effective operation of the Marine Parks.

Although the focus of the review is on the aspects below, it is not limited by them:

 The extent to which the objects of the Marine Parks Act have been delivered

 The extent to which the strategies of the Marine Parks Management Plans are appropriate

 The extent to which the strategies of the Marine Park Management Plans have been implemented, and their contribution to the delivery of the objects of the Marine Parks Act.

3.2 Program theory and program logic

This Framework adopts the program theory and program logic as defined by Markiewicz and Patrick (2016). Program theory makes explicit the reasoning as to how and why it is believed that the actions will produce intended results, whilst program logic is operational in nature in identifying an intentional and sequential progression from a program’s actions to its results over time.

The program theory is expressed as theory of changes identified in the baseline reports prepared for each marine park. These reports outline marine park values, the pressures that are impacting on the values, and the changes that are expected from the implementation of the management plan.

Conceptual diagrams assist with illustrating the theory of changes, which inform the identification of indicators to be monitored so the expected changes can be measured. They are a representation of the understanding of the systems, and are of primary importance in capturing and synthesising values, threats, processes as well as cause and effect relationships. The Baseline Reports adopt the Integrated Application Network (IAN) (University of Maryland, Center for Environmental Studies) (UMCES) process to document and record key ecosystem function and process assumptions. Conceptual diagrams are used as tools for communicating the Program Theory.

The program logic for the marine parks program is a dynamic visual management tool, which is

(17)

good practice planning, the logic model tries to establish a clear logic between the program’s

efforts/resources and its intended results through the identification of how and why program’s activities should lead to certain outcomes that reflect expected changes as per Program Theory. It helps the program managers to model how resources invested in the program and funding activities being undertaken, are understood to produce results. A key element of program logic is the ‘outcomes hierarchy’, which plots a chain of expected consequences arising from planned activities. Individual outcomes are mapped according to a timeframe over which they are anticipated to occur. Shorter term outcomes are linked to longer term outcomes by assumed cause-effect relationships. A program logic usually has the following main elements:

1. Inputs – resources available 2. Activities – specific actions taken 3. Outputs – products or services

4. Outcomes – changes in knowledge, skills, values (short term), behaviour (mid-term), condition (long term)

5. Assumptions – Statements or hypothesis that are believed to be true that link causation 6. External Factors – The environment outside the program that may have an impact on the success of the program

The assumptions underpinning the logic are defined as statements or hypotheses, which describe how and why it is believed that an outcome is achieved through cause and effect relationships. Articulating and testing assumptions is a process that contributes to learning and adaptive management; as such, it is a critical step for supporting the decision making process and is also of primary importance as an evidence-base or baseline from which to benchmark change in knowledge, resource condition and ecosystem health. Research projects will be funded in order to validate some of the assumptions on a needs basis. Assumptions are listed separately to the program logic, but relate to its steps.

External factors are another element of the program logic as they describe the influences external to the program, which may impact the achievement of desired outcomes. They are also listed separately to the program logic, but relate to its steps.

In the context of establishing baselines and a monitoring program for Marine Parks, conceptual models (a synthesis of key values and supporting scientific evidence base) will be undertaken at two scales:

1. Marine parks (19)

2. State bioregional network (1)

The program theory, conceptual models and associated program logic are dynamic elements that are refined as results of the monitoring program and research projects are produced. The objective is that they are informative at all stages of planning and adaptive to change as greater understanding about the theory of change and associated assumptions is achieved via monitoring and research.

The version of the Marine Parks Program Logic Model depicted the Appendix (as of the date of this report) highlights the activities, outputs and outcomes of the four sub-programs of the Marine Parks Program (Stewardship, Protection, Compliance and Performance) being implemented to deliver on the Strategies of the Marine Park Management Plans.

(18)

3.3 Baseline reports

The baseline reports summarises the available information and our current state of knowledge and historical trends for each of the 19 marine parks. The baseline reports inform the Marine Parks MER Program by providing a knowledge base, a conceptual understanding and predictions and indicators of change based upon the current knowledge of relationships between six components: ecological values, social and economic (socio-economic) values, physical drivers, socio-economic drivers, human-

mediated pressures and marine park management plans. The reports include an inventory of the available information with an emphasis on the nature of information, the scale (temporal and spatial) and indicators data sets and monitoring methods that could inform and also be used within the MER Program. The information from the baseline reports for each marine park will be aggregated to inform a conceptual understanding of the bioregional network the MER Program.

3.4 Key evaluation questions and subordinate evaluation questions

The evaluation of the effectiveness of Marine Parks’ Management Plans in delivering on the objects of the Act will be based on key evaluation questions, which will also inform the statutory review of the Management Plans. As such, the evaluation component of the MER program will focus on answering key evaluation questions as a way of meeting the requirements of the MER program. The purpose of setting key evaluation questions is also to provide direction to monitoring and evaluation activities consistent with outcomes of the program logic. The Marine Parks Executive Committee for the Marine Parks Program has endorsed the following key evaluation questions based on the objects of the Act:

1. To what extent has the legislated comprehensive, adequate, representative (CAR) system protected and conserved marine biological diversity and marine habitats?

2. To what extent have marine parks contributed to the maintenance of ecological processes in the marine environment?

3. To what extent have marine parks contributed to the adaptation to the impacts of climate change in the marine environment?

4. To what extent have the marine parks contributed to protecting and conserving features of natural or cultural heritage significance?

5. To what extent have the marine parks contributed to allowing ecologically sustainable development and use of marine environments?

6. To what extent have the marine parks contributed to providing opportunities for public appreciation, education, understanding and enjoyment of marine environments?

To determine the effectiveness of the program in delivering the objects of the act these evaluation questions should seek to collectively address the following criteria:

1. Program planning and design – Assessing the appropriateness of the design

2. Program objectives – Assessing program effectiveness in meeting its objectives, its value and quality

3. Program implementation – Examining efficiency in program implementation

(19)

4. Program Results – Establishing the impact the degree to which the program is attributable to the change

5. Sustainability of results – Identifying ongoing sustainable benefits of the program.

These overarching questions will be broken down into more specific subordinate evaluation questions and operational components, such as assumptions, monitoring indicators and methods, measures, evaluation approach and scale of implementation, which will be documented within the MER Plan.

3.4.1 Prioritisation of the key evaluation questions and their subordinates

Following the development of KEQs and their subordinate questions a prioritisation process will be undertaken to ensure that the resources provided to the MER Program addresses the key questions and assumptions. The initial framing of this process is ‘what is the risk to the outcomes of the program by not answering this question’. Additional factors to be considered include the scale of issue, the level of comprehensiveness required (spatial and temporal), the accuracy or level of scientific supporting evidence required, the need for a targeted approach and the cost-effectiveness of the investment in relation to the Marine Parks Program. This prioritisation process is also useful in providing a transparent method and process for rationalising why certain questions and activities would not be supported or invested in.

3.5 Monitoring, Evaluation and Reporting Plan

The critical component of the MER Program (Strategy 10) is the MER Plan, which outlines the ‘what, where, why and how’ of the MER program. The MER Program is guided by an adaptive management framework which aims to continually inform and improve the management of marine parks. The MER Plan describes the monitoring, evaluation and reporting aspects of the program. It is informed by the outcomes of the previous Steps 1 to 4 (which should be incorporated in the plan); including the

scientific rationale, referring to supporting information and evidence from the baseline reports; and the strategic links to the operational plans. The MER plan has four sections, including timing and scope, as described below:

3.5.1 Monitoring

This section describes how research and monitoring will contribute to answering the evaluation questions. The monitoring section provides a guide to the ongoing and systematic collection and analysis of routine performance information, refers to data collection protocols, highlights the progress of implementation and identifies results being produced with focus on the key evaluation questions. It includes a description of performance indicators, methods and measures; data collection and tools;

responsibilities and timeframes.

As described above, the purpose of the MER Plan is to determine what questions need to be assessed rather than have the question defined or limited by the indicators and data currently available.

3.5.2 Evaluation

This section focusses on assessing whether the program is achieving its intended results, what works or doesn’t work well and why, the program’s quality and value, and the extent to which it meets the expectations of delivering on the objects of the Marine Parks Act. The monitoring activities identifies

(20)

the what and where, whilst the evaluation process focusses on the why and how. The evaluation process determines which evaluation methods and survey designs are most appropriate to provide fit- for-purpose answers with regards to the questions and supporting data and information. It points out the focus of the evaluation, evaluation methods and methods for implementation for each evaluation question, protocols for evaluating the data, responsibilities and timeframes.

3.5.3 Reporting

This section focuses on how the data and information collected, results and learnings of the monitoring and evaluation meet the requirements of internal and external audiences as part of the accountability and responsibility that a program has in relation to its progress and performance. The assessment about the type of communication products and who and when the information will be disseminated will be addressed in more detail within the communications plan.

The reporting section includes details on the main information products and their format, (e.g. annual status reports, mid-term implementation assessment, end term evaluation of effectiveness and review of the management plans.) the audiences for distribution and their requirements, responsibilities and key timeframes. Reports that address key evaluation questions, include valid and reliable performance information that highlights accomplishments against expected results and other performance, and that also demonstrates a capacity to learn and adapt. Table 2 (Markiewicz and Patrick, 2016) provides guidelines for an indicative evaluation report structure that can be adapted and developed to address the reporting requirements of the Marine Parks MER program.

(21)

Table 2. Indicative MER report structure (reprinted from Markiewicz and Patrick, 2016)

Chapter headings Considerations

Summary

1. Program overview

- Context and background to the program - Approach to monitoring and evaluation

- Presents the context and history of the program

- Identifies the purpose and the approach adopted to monitoring and evaluation - Outlines approach to key stakeholder engagement

2. Foundations

- Program Theory / Logic - Evaluation questions

- Program theory / logic outlining expected changes and results

- Evaluation questions scoped with stakeholders and priorities agreed to by stakeholders (data availability and resourcing)

3. Methodology

- Approach to monitoring - Evaluation methodology

- Scope of the approach and methodology used

- Any limitations or constraints in approach or methodology used - Any ethical issues that arose and how they were handled 4. Key results

- Program context (appropriateness) - Progress towards objectives (effectiveness)

- Program implementation and resourcing (efficiency)

- Program coordination and management (efficiency)

- Program outcomes (impact); Progress against program logic and assumptions - Sustainability

-Overall evaluative conclusions

- Answers provided to the evaluation questions under each domain

- Presents ssynthesized data, assessments and findings against each evaluation question

- Identifies performance against indicators and targets, criteria and standards - Presents an assessment of program impact

- Provides an assessment of progress against the program theory/logic - Assessment of validity of key assumptions

- Assessment of other areas examined, such as environmental impact

5. Recommendations - Focused on program continuation and/or improvement

6. Learning - Identifies lessons that can be used to review program design and benefit future program development and policy context

7. Appendices

- Data collection tools and approaches - Performance indicators and targets, evaluation and other analytical frameworks

- Presents range of data collection tools used - Identifies approaches used to sample -Lists consultation groups / interviewees

- Identifies ethical approval processes and informed consent forms - Presents approaches for data synthesis and assessment, such as use of performance indicators and targets, evaluation and other analytical frameworks

(22)

3.5.3 Managing Environmental Knowledge

The MER Plan will outline how the Marine Parks Program manages the data and information in accordance with Government of South Australia and DEWNR requirements through alignment with State Records Act 1997, Information Management Framework (IMF) and Monitoring Environmental Knowledge guidelines (MER). These principles have been developed by DEWNR and adapted from whole of government frameworks and are designed to be broad and enduring.

Under the Information Management Framework, DEWNR has established overarching rationale, principles and lifecycle procedures that the Marine Parks Program will adhere to. These information principles guide the MER programs approach to the management, use, sharing and investment in data and information (see Table 3).

Table 3. Information management principles

Principle Meaning

Valued Information is recognised as a valuable asset that government holds and manages in trust on behalf of citizens.

Shared Information is openly shared, proactively released and licensed to promote re-use.

Information is not the property of an individual or agency but is held by the Crown in the right of the State of South Australia.

Trusted Information is accurate, relevant, timely, available and secure.

Managed Information is actively planned and managed throughout its lifecycle.

Applied Information is proactively used and provides the evidence base for our decision making.

In addition, the Marine Parks Program will follow the Managing Environmental Knowledge Procedure a procedure developed to manage environmental data and knowledge for enduring use, according to corporate standards and protocols and guided by both the DEWNR Information Management

Framework and the government’s Declaration of Open Data. The four primary components of this are listed below and shown in Figure 2:

1. Data management on Network Systems (all formats)

2. Document/records management on the DEWNR internal network (iShare) 3. Communications management for internal and external audiences

4. Authorisation and publishing procedures.

(23)

Figure 2. Summary diagram of DEWNR network folders and websites

Environmental knowledge includes (but is not limited to) biological, hydrological, geophysical and natural resource management information held in any format. Information derived from biodiversity conservation activities, land management activities, biophysical, social and economic research, landscape planning and policy activities.

3.6 Operational planning

3.6.1 Marine Parks Operational Plans

The monitoring requirements for the Marine Parks Program is informed by the MER Plan and are operationalized through the Annual Marine Parks Project Plan planning process undertaken by each sub-program (Stewardship, Compliance, Protection and Performance). This ensures activities,

resourcing and monitoring activities are committed to on an annual basis to deliver on the reporting requirements.

Management activities must be monitored across the marine parks program to enable evaluation of the effectiveness of the management plans and to assist with interpretation of monitoring data on

ecological and socio-economic values and management activities.

Linked to the operational plans are the annual monitoring reports describing the yearly monitoring activities. These should include on an annual basis Ecological Monitoring Status Reports, Socio- Economic Monitoring Status Reports, Compliance Reports, and Stewardship and Protection status reports.

(24)

3.6.2 Research Strategy

The Marine Parks Research Program is complementary to the MER Program, it aims to both build partnerships with the research community and provide an additional and external source of skills and experience to inform the management of Marine Parks. It will contribute data and information to address the key evaluation questions, fill key knowledge gaps and improve the knowledge base of the Marine Parks.

Research outcomes could inform the evaluation of the effectiveness of marine park management plans and management decisions, test the assumptions that underpin the predicted changes, identify cost effective and feasible monitoring methods to improve the monitoring, evaluation and reporting program and the marine parks program overall.

The primary aims of the research program outcomes are:

1. Research will contribute to informing the key evaluation questions of the marine parks performance program

2. Research will significantly enhance our understanding of South Australia’s marine biodiversity in marine parks.

Potential research projects will be aligned with one or more themes:

1. Ecological systems: status and processes

2. Communities: social, cultural and economic values and assets 3. Management effectiveness

The MER Plan will identify knowledge gaps and evaluation questions that are best approached through research. This will inform a Marine Parks Research strategy and Priorities Register to support ongoing research and guide new research.

3.7 Communications Plan

The Marine Parks Communications Strategy is an overarching document that provides direction to the program and the Marine Parks Performance Communications Plan. This aligns with the Marine Parks Act 2007 Objective 8(a)(v) “to assist in providing for public appreciation, education, understanding and enjoyment of marine environments’. In addition, effective communication assists in the delivery of the following four Marine Park strategies as described in the 19 Marine Park Management Plans to:

1. Provide for public appreciation, understanding and enjoyment of the marine park (Strategy 5).

2. Provide education to support the implementation of the marine park (Strategy 7).

3. Seek to involve local communities and stakeholders in the day-to-day management and monitoring of the marine park (Strategy 8).

4. Ensure outcomes of the MER Program and research outcomes are made publicly available and inform decision-making and periodic review of this management plan (Strategy 12).

And, further assist in fulfilling the Government of South Australia’s Marine Park election commitment to

“Develop educational materials to promote the results of the monitoring program”.

(25)

The Marine Park Performance Communication Plan provides guidance on communicating with key stakeholders about the outcomes of the Marine Parks MER program (Strategy 12).

The Marine Park Performance Communication Plan will identify:

 key audiences, what they want to know, and when and how the information will be provided and delivery platforms;

 information delegations; media spokes people and subject matter experts

 implementation planning including, known program reporting outputs; outcomes and timing.

Specifically describe the course of action to:

 inform stakeholders about MER program work undertaken i.e. findings, outcomes, activities, reports, partnerships and opportunities

 update stakeholders about progress of the MER Program and marine parks performance monitoring trends and outcomes

 improve public awareness about marine parks economic endeavors and community benefits.

3.8 Implementation Schedule

The implementation of the MER framework includes clear identification of the tasks, timelines and responsibilities for all the steps identified in the framework. Due to the Marine Park Program (MPP) complexities of internal and external delivery partners’ schedules and requirements only the fixed high level deliverables will be discussed in the MER Plan. The Implementation Schedule will outline all the program delivery timelines and milestones, including products and communications that are

components of the MER and Communications plan. Therefore, the implementation schedule will require periodic review and updates to program logic and adaptive management response to program is implemented.

It is envisaged that the program schedule will be in the form of a Gantt chart for internal use and accessed through internal communications tools such as the DEWNR network site (iShare).

Development of the implementation schedule considers activities that are:

1. Internal to MPP program:

a. Marine Parks Performance Program requirements for delivery b. Research Strategy project delivery and incorporation of outcomes 2. External to MPP program:

a. Marine Parks Program requirements for delivery (Compliance, Stewardship, Protection and regional delivery)

b. The process for review of the Marine Parks Management Plans 3. Communication and adaptive management in general:

a. The timing and frequency of reporting required

b. The timing and frequency of communication products to multiple stakeholders.

(26)

4 Appendix. Program Logic Model for South Australian Marine Parks Program

*Note: This Marine Parks Program Logic Model is a component of an ongoing adaptive management approach and therefore some of this content will change as the program develops. This is model version #4, produced September 2016.

Long term outcomes (20+ years)

Assisted in the maintenance of ecological processes

Assisted increased adaptive capacity to the

impacts of climate change

Provided for public appreciation, education, understanding and

enjoyment Assisted in allowing ecologically

sustainable development and use Assisted with the protection and

conservation of features of natural or cultural heritage

significance

Healthy, Diverse and Productive Marine Environment for Future Generations

Marine biological diversity and marine habitats are protected and conserved by a CAR

system of marine parks Aspirational Goal

Intermediate Outcomes (10 years)

Immediate Outcomes (5 years)

1 Assisted in mitigating threats to marine biodiversity and marine habitats by actively influencing activities and uses within and adjacent Marne Parks

2 Reduced impacts on marine ecosystems through management of activities and uses in accordance with zoning and special purpose area provisions.

3 Enhanced or maintained ecosystem status, function, connectivity and resilience for better ecosystem service.

4 Enhanced or maintained biological functions in SZ.

5 Increased number and size of some marine species in SZ.

6 Enhanced or maintained marine habitats.

7 Maintained or improved the condition of natural and cultural heritage assets.

8 Sustainable development and use of marine resources supported.

9 Appropriate activities are lawfully conducted in accordance with Marine Parks Regulations and Management Plans

10 The implications of climate change on marine ecosystem drivers understood.

11 Capacity of marine ecosystems in to adapt to climate change increased.

12 Local communities and stakeholders involved in the day-to-day management of Marine Parks.

13 Increased number of people valuing and understanding Marine Parks

14 Country, plants, animals and culture conserved through cooperation with Aboriginal communities.

15 Improved or stable tourism

16 Improved or stable aquaculture, fisheries, mining, shipping and regional economies.

17 Effectiveness of marine park management plans and its contribution to the network measured

18 Local communities and stakeholders involved in the day-to-day monitoring of Marine Parks.

19 Management Plans review informed by the results of the evaluation of their effectiveness, and updated

knowledge/understanding

20 Performance of Marine Parks properly and effectively communicated

21 Voluntary compliance maximised 22 Effective deterrents created

1. The Marine Parks Act 2007 and supporting Regulations are effectively administered and implemented.

2. Advice on activities and uses within and adjacent to the marine park provided to government, stakeholders and community

3. Permit Regulations implemented and any permits issued support achievement of management plan objectives 4. Enhanced or maintained recruitment of marine species in

SZ.

5. Reduced disturbance in marine life and habitats in SZ.

6. Reduced impacts of nutrients, sediments and pollutants, from all sources, on Marine Parks

7. Relevant development plans consistent with MP management plans.

8. Fisheries not impacted by more than 5% GVP (completed 2014).

9. Opportunities for sustainable nature-based tourism in marine parks created and promoted.

10. Implementation of marine parks supported through education

11 Increased awareness and understanding of Marine Parks among South Australian Community

12. Increased positive media

13. Improved understanding of sanctuary zones and their habitats

14. Opportunities for local and Aboriginal community monitoring projects created.

15. MER program developed and implemented

16. Partnerships with community and stakeholder involvement to implement the MER program fostered and opportunities for community and stakeholder involvement incorporated 17. Outcomes of research and the MER program are made

publicly available and informed decision-making and periodic review of management plans.

18. Integrity of knowledge frameworks that underpin predicted outcomes assessed through priority research and fostered partnership

19. Media and other opportunities used to promote MP performance

20. Findings and resources shared with community

21 Monitoring plan, based on conceptual models and baseline report, implemented

22. Compliance strategy implemented

23. Measures to address serious or repeat non-compliance established

24. Appropriate enforcement options used at priority sites for priority issues

25. Success of permit system increased

26. Understanding of the values of marine environment and Marine Parks encouraged

27. Apps, maps and gps coordinates improved 28. Number of trained wardens increased

29. Compliance is supported by an across Government collaboration 30. Compliance and enforcement activities reviewed and improved 31. Sanctuary Zones are monitored beyond effort and incident and

intelligence recording explored.

framework for Marine Parks Program

Monitoring, Evaluation and Reporting