Assessing policy success and failure – Testing and adapting an environmental policy evaluation framework in rural China

Abstract

China does not have a systematic and holistic framework for evaluating environmental policy and programs. The absence of effective evaluations of the Comprehensive Environmental Treatment (CET) policy, which focused on comprehensively addressing rural waste management and pollution control in rural areas, is an example of the lack of a systematic and holistic policy evaluation framework in China. Such a framework is needed given the range of problems faced and initiatives undertaken, and this is the gap this study aims to address. Based on a review of the policy evaluation framework literature, McConnell’s policy evaluation framework was identified as a potential framework for application in the Chinese context. However, this framework has not been thoroughly tested. An integrated approach was taken to explore and test McConnell’s framework in the Chinese context. This approach included three main and complementary methodologies: adaptive learning, case studies, and triangulation. A variety of qualitative and quantitative methods were used to examine four programs implementing the CET policy in rural China. McConnell’s framework was first evaluated from a theoretical point of view to lay the foundation for further testing and potential revision. A guideline for measuring degrees of success/failure of the criteria across three realms (process, program, and politics) was developed. McConnell’s framework was then tested, and improved in light of emergent knowledge, against the CET environmental policy programs, in two phases each involving two case applications. Findings first showed that McConnell’s framework and most of its criteria are applicable in the Chinese context, subject to some relatively minor but important modifications. Second, there were various levels of policy success and failure between cases and across realms. Third, the evaluation results were compared, showing that delegating particular areas of authority to local governments and communities, and delegating relevant decision-making power to more actors, could help to improve the effectiveness of a policy. Failures in policy shaping do not necessarily result in policy implementation failure, and policy formulation and implementation are related to the political impact of a policy. Fourth, there are multiple and complex interactions between individual behaviour, institutional design, and prevailing societal values which contributed to different degrees of policy failure. This research indicates that the amended McConnell framework and the methods developed in this research are practical and usable for evaluating environmental policy in the Chinese political context. Especially at a higher policy level, use of the amended McConnell framework can provide a holistic view of the policy process for the identification of bigger picture, more strategic, questions. The amended framework can also provide a meaningful complement to the existing CET policy evaluation method. This research further deepens the understanding of policy success and failure, thus providing an additional reference point for future research.