The Laboratory for Political and Social Analyses (LAPS, Laboratorio Analisi Politiche e Sociali – website: www.laps.unisi.it) is a centre of excellence for scholars interested in public opinion research, providing them the equipment and expertise to undertake face-to-face, telephone, online and mixed-mode surveys, organise participatory and deliberative events, perform high-level analysis, and prepare reports on social, political and economic issues. LAPS offers expertise in questionnaire design and evaluation, data collection, and data analysis. Our research team provided scientific advice to several international projects, such as the Transatlantic Trend Survey, the Transatlantic Trends Survey on Immigration and the European Elite Survey.
The research team is composed by Prof. Pierangelo Isernia, Director and senior advisor, Dr. Francesco Olmastroni, Dr. Rossella Borri and Mrs. Manuela Maltese.
The range of research activities carried out at LAPS goes from local issues to (inter)national politics and society.

 

LAPS uses for its research activities IdSurvey software.
IdSurvey provides for extreme flexibility in programming complex questionnaires and experimental designs conducted in CATI, CAWI, CAPI or mixed modes. In particular, we find the following functionalities very useful:

  • The organisation of random and stratified sampling frames based on an unlimited number of parameters as well as the automatic update of quotas based on sample information, answers given to specific questions included in the questionnaire or a combination of the two.
  • The possibility to create general or specific filters for interviewers, contacts, questions, and answer options.
  • The wide range of questions that can be created.
  • The possibility to randomize all elements of the questionnaire: single pages or groups/blocks of pages, questions, answer options, etc.
  • The possibility to display answers or information about the contacts during the questionnaire.
  • The possibility to export the collected data in different formats: SPSS, CSV, Access and Excel.

One of the most challenging projects we were involved in has been the Policy Delphi. Carried out as part of the TRANSWORLD project (Project no. 290454), funded by the EU under the 7th Framework Programme, it focused on the analysis of transatlantic relations and the future of global governance. Within this framework, the Delphi was designed to primarily explore the following aspects:

  • Whether and to what extent a set of global challenges are likely to emerge in the four areas addressed by the project (i.e., international security, global economy, global environment and climate change, human rights and democracy promotion) and what would be their impact, should they occur.
  • What policy responses to such challenges are desirable and/or feasible.
  • What is the state of governance in each of the four areas and how it is changing.

The final results of the Delphi provided a solid ground for the development of policy recommendations to decision-makers, in order to foster the viability, effectiveness, and accountability of global/regional governance architectures.
The Delphi technique is a forecasting “method for the systematic solicitation of informed judgments on a particular topic”. It is a tool to analyse complex policy issues and to provide informed judgments about the estimated impact of policy decisions.

The TRANSWORLD Policy Delphi has been designed, coordinated and conducted by a research team of the Department of Political, Social and Cognitive Sciences at the University of Siena, which was also responsible for data collection and analysis.
IdSurvey has been used for the implementation of the entire process, which has been articulated in different stages:
Experts’ selection and recruitment (April – June 2014)
Round 1 (October 2014-January 2015): “Exploring challenges and policy solutions”
Round 2 (January 2015- March 2015): “Reaching consensus and identifying areas of polarisation”
Round 3 (May 2015- July 2015): “Towards policy recommendations”

 

Experts’ selection and recruitment (April – June 2014)
The TRANSWORLD Delphi was designed to survey around 30 experts for each of the four areas of research, namely international security, global economy, global environment and climate change, human rights and democracy promotion.
Since April 2014, a list of around 300 experts’ names from the EU, the US, Turkey, Canada, Brazil, Russia, India and China was collected. They were then ranked according to a number of criteria, such as:
a) scope of their expertise (international/regional vs. country-based approach, with preference given to the former);
b) seniority (with preference given to senior experts, where ‘senior’ is defined as experience exceeding 17 years, beginning with their first post-graduate or post-doctoral job);
c) number and quality of publications (with preference given to those who published relevant articles on the policy area of expertise).

Experts were further classified according to their role, which could be: academics, think tankers, members of non-governmental organisations (NGOs), practitioners.
The study was designed to be representative with regards to country and role.
Accordingly, 266 experts (from 53 to 77 for each area) were shortlisted and formally invited to participate in the exercise between July and October 2014. Table 1 below shows the response rate (number of experts that accepted divided by the total number of invited people) in each of the four domains.

 

International security

Global Economy

Global Environment and Climate change

Human Rights and Democracy Promotion

A Contacted

53

73

77

63

B Accepted the invitation

32

30

29

31

B/A Response rate invitations

60%

41%

38%

49%

C Filled survey Round 1

24

21

26

25

C/B Response rate Round 1

75%

70%

90%

81%

D Filled survey Round 2

24

20

26

23

D/C Response rate Round 2

100%

95%

100%

92%

E Filled survey Round 3

24

19

26

22

E/D Response rate Round 3

100%

95%

100%

96%

 

Round 1 (October 2014-January 2015): “Exploring challenges and policy solutions”
The first round explored experts’ opinions about the main challenges and policy solutions in the four areas under study.

Experts were asked:

1) to list the main challenges in their field of expertise in terms of both impact and likelihood, on a scale from 0 to 100;
2) to express their opinion about the desirability and feasibility of a list of policy solutions to address them, on a scale from 1 to 10;
3) to express their opinion about a set of questions, which were intended to collect information about salient issues concerning the area of concern, on a Likert scale (agree/disagree options);
4) to suggest further challenges and solutions not included in the questionnaire, through comments.
The four questionnaires for the first round (one for each policy domain) have been drafted by relying on the research effort conducted under the broader TRANSWORLD project. Accordingly, working papers, reports and other research material produced by the TRANSWORLD project were analysed, in order to identify and list the core issues and themes to be addressed by the Delphi exercise.
Each questionnaire was preliminary discussed with scientific advisors and pilot versions of the questionnaires were delivered to a number of scholars to evaluate their length, understandability and clarity.
The final versions of Round 1 questionnaires were administered to the experts by the end of October 2014 in CAWI (Computer Assisted Web Interview) mode. Each expert received a unique, personal link to the questionnaire related to his/her own field of expertise. They were given a deadline of two weeks to complete the survey; when necessary, reminders were sent after the deadline expired, in order to increase the response rate. The response rate for Round 1, accounting for those experts who filled the survey, ranged from 70% for Global Economy to 90% for Global Environment and Climate Change (see Table 1).

 

Round 2 (January 2015- March 2015): “Reaching consensus and identifying areas of polarisation”
The second round identified the areas of consensus and those where disagreement existed over the main challenges and solutions. In the questionnaire of Round 2, items were clustered into thematic groups, based on factor analysis and grounded in the relevant theoretical literature. Experts were then presented with the results of the previous round for each item, reported as average values and standard deviations and introduced by a short commentary. In this way, experts were fed back with the information to evaluate the amount of consensus or disagreement within the group on each item. They could also see on the screen their own responses in the first round. Then, in light of the answers of the group, they were invited to either confirm or revise their assessment. They were also strongly encouraged to comment on their personal assessments. Furthermore, based on the comments of the panellists in Round 1, a few new items and/or modified some old ones were introduced in the second round questionnaire.
In March 2015 the four questionnaires were sent out for completion to experts, with a deadline of two weeks to complete the survey. After three weeks we reached the 90-100% of response rate in all the four areas (Table 1).

 

Round 3 (May 2015- July 2015): “Towards policy recommendations”

This round was designed to draw conclusions about near- and long-term future challenges to the international security, global economy, global environment and climate change, human rights and democracy promotion, and the solutions available to address them.
Experts received a summary report of the main findings and charts, based on data of the previous round, and asked to provide a commentary on their panel’s estimates. Only for those items where a large disagreement still existed, they were asked once again to either confirm or revise their judgments and to specify the reasons for their position.
A few new questions were introduced to verify results, explain controversial issues, and test general concepts in more concrete scenarios.
Below there is an example of the output visible on the screen, taken from the third round. In Figure 1 there is the introduction to survey, where the task was briefly explained.
Following figures illustrate the case of a reiterated question: experts were preliminarily presented with the summary of their panel’s responses in the previous round (Figure 2). Then, the critical item where consensus was not reached was asked again, while the expert could see panel’s mean response and the standard deviation (in green). In the box below, he/she could see his/her previous response and he/she could change or confirm it. In Round 3, comments to new responses were compulsory.

 

Figure 1. Example of introduction to survey in Round 3

 

Figure 2. Summary of responses of previous round

 

Figure 3. New response to controversial item

Get your free e-book

Insert your name and email to receive the link to download the free ebook.
You won't receive any spam. We hate it as much as you do! 🙂

Do you want to know more?

Download your e-book now for free.

The essential guide to CATI CAWI CAPI methodologies

You will also get:
“The essential guide to CATI CAWI CAPI methodologies” ebook
Access to exclusive material
Access to videos and webinars
The opportunity to request a free consulting session
In a few minutes you’ll receive your exclusive ebook to improve survey quality and reduce costs.
You have already voted.
Free ebookThe essential guide to CATI CAWI CAPI methodologies.

Free ebook
The essential guide to CATI CAWI CAPI methodologies.

Insert your name, your email and download your free ebook now!

You have Successfully Subscribed!