Evaluating OpenCoesione

The following offers a look at the main results of evaluations of the OpenCoesione initiative:

 

Evaluation of the first three years of the project

In 2015, our third year of operation as we were preparing to formally become Italy’s single website for the 2014-2020 cohesion policy cycle (as ratified by the Partnership Agreement and based on the provisions of Article 115 of Regulation (EC) no. 1303/2013), OpenCoesione issued a call for tenders for a first assessment of the initiative.

This evaluation was aimed at understanding whether OpenCoesione had been of value to government and helped to improve cohesion policies by focusing, in particular, on uses, on the characterisation of users, and on the way the website is used, both within government and outside.

The evaluation was conducted by Contesti srl.

The Summary Assessment Report presents the analysis and provides initial answers to the evaluation questions, while also providing a number of suggestions for the future of the portal.

In order to achieve the objectives of the study, a variety of tools and approaches were adopted as described in the report The Users-Needs Map and Research Approach. The report OpenCoesione on the Web presents the results of the analysis of website statistics. The reports The Opinions of OpenCoesione Users and The Characterisation of OpenCoesione Users provide the results of the empirical studies based on a survey of current and potential users of the website, while the report The Views of Key Users presents the results of a series of interviews of a select group of key users. In addition to these traditional research methods, a number of unconventional approaches were also used, such as web scraping and sentiment analysis.

The annex on methodology also provides further information on the two empirical studies conducted, for which the datasets (in CSV format) of the survey questionnaires and their corresponding metadata files are also available:

survey of actual users

survey of potential users

 

Who uses cohesion policy open data, how and why

In 2014, within the scope of the NUVAL Project “Actions to Support the Efforts of the National Evaluation System and the Evaluation Units”, which was co-financed by the Governance and Technical Assistance NOP 2007-2013 (EDRF), a research grant for the analysis and evaluation of development policies and public investment was financed.

The grant was awarded to Michele Lo Russo (who has a PhD in Comparative and European Politics from the University of Siena) and led to the publication of a study entitled “Who uses cohesion policy open data, how and why”was part of the Materiali UVAL series.

The study proposes a metric for evaluating the strategy of releasing open data based on transparency, participation, and efficiency/innovation as mechanisms for creating value, and it features a qualitative and quantitative analysis of OpenCoesione users.

Through an analysis of traffic on the OpenCoesione website, the scope of demand, the types of user, and the makeup of the various user bases is established. This analysis suggests that OpenCoesione primarily creates value through transparency, while only partially through participation and collaboration and even less through efficiency and innovation. In addition, an online survey of users from the various user bases identified as result of the traffic analysis made it possible to determine how much individual and collective factors of the demand for information influence these results. The study showed that user motivations and capabilities, and not their perception of the opportunities that the data presents, are the factors that have had a significant impact on the types of use of the data and on the mechanisms of value creation activated. Finally, the study presents a number of possible actions to strengthen the OpenCoesione release strategy.