Part 5: How to adapt the questionnaire

5.1 Finding the right balance when adapting the questionnaire

This part covers how to test the existing Innovation Barometer questionnaire in  your national context, select what questions to ask and how to translate pre-existing questions. You may also find this section useful if you test and adapt questionnaires you have found elsewhere.

Adapting questionnaires involves striking a balance between only making a few changes to ensure international comparability and managing your resources on the one hand, and adapting it to fit your strategic, national and cultural context on the other.

The Innovation Barometer questionnaire is presented in Appendix 2 with indications of selected core questions that are recommended if international comparability with existing data is a priority.

ACTIONABLE ADVICE 5.1
TRUST THAT YOU ARE THE EXPERT ON WHAT CHANGES YOU NEED TO MAKE

The Copenhagen Manual cannot provide definitive answers on how much you need to adapt the questionnaire. You are the expert on your country and are the best person to decide on what adaptations are needed. Try to achieve a balance between necessary changes and changes that would be nice. Consider which questions from existing Innovation Barometers you want to be able to compare your data with – and use this to guide your changes. Significant changes might be necessary if adapting the questionnaire to a country that differs greatly from the Nordic countries, where the Innovation Barometer was first developed.

USE CASE
THE NORDIC INNOVATION BAROMETER

What does public sector innovation look like in the context of public schools, libraries and hospitals in the Nordic countries? How innovative are the Nordic public sectors, and who collaborates with public sector workplaces on innovation?

The answers to these questions were provided by the nine organisations responsible for the five Nordic Innovation Barometers when they joined forces in 2019 to present key results from the Innovation Barometer for each of their countries in a joint publication.

Results were astonishingly similar, with four out of five public sector workplaces in all countries introducing at least one innovation over a two-year period. Similarly, four out of five innovations were carried out in collaboration with external partners, e.g. private companies, citizens, non-governmental organisations or other public sector workplaces.

The Nordic countries were able to present the results in this manner because they all used a similar approach. The overall definition of public sector innovation is close to identical in the five countries and many questions are the same. However, some results were challenging to present as some answer categories varied and not all of the questions presented were asked in all of the countries.

The Nordic comparisons shed new light on national results by adding the context of other Nordic countries, sparking new national and international interest in the results.

5.2 Testing the questionnaire

The most important part of adapting the questionnaire to your local context is to test the questions and the questionnaire with public sector employees who are not innovation specialists. To use your resources efficiently, start by having a few people (e.g. co-workers, networks and family) look at a rough trans- lation of an existing Innovation Barometer questionnaire (innovationbarometer.org). See if they understand the questions and if they would be able to answer them. In addition, ask if the answers to these questions would be relevant in the context of their respective workplaces. This quick feedback can provide a great starting point for selecting what questions to ask, adapting the questions to your local context and fine-tuning the translation of the original material.

WARNING!

Do not get in your own way. Make the respondent opinions and feedback on the questions the front and centre of your attention in the ongoing development of your questionnaire. To put it bluntly: Respondent opinions on the questions are more important than your own ideas about what the questions should be like. If respondents do not understand the questions as intended, the results will be of little or no use.

Once the questionnaire has been adapted, doing a systematic pilot study is also recommended. In a pilot, the draft questionnaire is tested on a small sample of respondents or a group of people highly similar to your respondents. Their qualitative feedback can be used to make changes to questions, and the quantitative results will provide an idea of what the final results may look like – giving you the opportunity to make changes if it becomes evident that something is missing or not working. A pilot will also help to test that the survey software for contacting respondents and collecting answers is working as expected. If you are not familiar with survey methodology, collaborate with someone knowledgeable on how to conduct a pilot study. Figure 5.1 provides a visual summary of the steps involved in testing the questionnaire. 

Any testing is better than no testing. If you do not have the resources to make a formal pilot, make sure to prioritise informal testing of the final questionnaire.

Figure 5.1. Steps involved in testing the Innovation Barometer questionnaire.

USE CASE
PILOTING IN COLLABORATION

In New Zealand, Creative HQ’s upcoming NZ Innovation Barometer pilot will provide senior public leaders with interactive data highlighting their agency’s strengths and areas for improvement. As the barometer is intended for organisational learning, the data collection method will be appropriately conducted in collaboration with the human resources departments of the participating agencies. These departments will assist both in designing questionnaires likely to engage staff and in executing a random sampling approach intended to reach 10–50% of staff through a one-off 30-minute survey.

Creative HQ has partnered with Victoria University of Wellington to design the survey and to develop the data collection method to ensure academic rigour and adherence to government data security standards.

In addition the pilot has been endorsed by the New Zealand’s Digital Government Leadership Group, which comprises twelve central government agency chief executives. This mandate provides credibility and relevance in advance to the data collection that pilot studies typically otherwise need to earn.

5.3 Selecting questions

Depending on your strategic goals (as discussed in Part 1) for the Innovation Barometer and your national context, some questions might be of more or less importance than others. Questions can be dropped, added and modified to fit your needs, but be aware that modifying and developing new questions will add more work and limit international comparability. Keep your strategy and audience in mind when making these decisions.

ACTIONABLE ADVICE 5.2 
THINK TWICE BEFORE MODIFYING QUESTIONS AND ANSWER CATEGORIES

Questions can be modified in multiple ways. Minor modifications include changing the exact wording to make it more comprehensible while keeping the meaning intact. It is often necessary to modify answer categories, as they may not fit your national context. For example, there might be specific national programmes for financing innovation that you want to add as an answer category to a question on funding innovation. Existing answer categories might, on the other hand, be irrelevant to your respondents and can be omitted. Use the testing of the questionnaire to decide what answer categories are relevant to your respondents.

WARNING!

International comparability will often be preserved with minor modifications like these, but larger modifications can com- promise it. Larger modifications include asking questions on the same subject but changing their meaning and the under- lying logic of the answer categories to such an extent that comparing the answers directly is no longer meaningful. Keep the comparability of questions in mind when deciding on what modifications to make.

You might want to omit questions that are not relevant. Dropping irrelevant questions will generally reduce the burden on respondents as filling out the questionnaire will not take as long.

It is strongly recommended that you keep some core questions to permit international comparisons. In figure 5.2 below, the light green boxes contain the core questions.

The questionnaire, which is available for download on innovationbarometer.org, also includes the full questions and indicates core questions with the same colour. Keeping the core questions intact increases the chance of making relevant interna- tional comparisons.

Keep in mind that some questions are logically necessary in order to ask and/or analyse other questions. For example, respondents in workplaces with no successful innovations must be identified if you want to analyse whether their organisational culture differs from workplaces with successful innovations.

Be sure to ask the core questions in the same order as described in figure 5.2 because it influences how respondents answer.

Figure 5.2. Flow chart of core questions.

ACTIONABLE ADVICE 5.4
DEVELOP NEW QUESTIONS IF NECESSARY

Depending on your strategy, you might want to add additional questions on themes that are important to you.

• If new questions are developed, testing them on potential respondents is essential.
• Strive not to make the questionnaire too long. For each new question added, a large number of respondents will have to spend time answering it, increasing the risk that they will give up on answering the survey.
• If multiple themes and a large number of new questions are important to you, consid- er focusing on certain themes and saving others for the next round of data collection.
• Limit the number of respondents who have to answer additional questions by conditioning questions on other questions when applicable. For example, respondents whose workplaces have not evaluated their latest innovation need not answer further questions on how they conducted the evaluation.
• It is best to ask your additional questions on the latest innovation in continuation of the other questions on this topic. Additional questions of a completely different nature should preferably be placed last to avoid influencing how respondents reply to core questions.

Remember to share your new or adapted questions with the Innovation Barometer community, as other countries might want to reuse them. Fast track the availability of your public sector innovation questionnaire to an international audience by submitting it to coi@coi.dk for rapid publication on innovationbarometer.org.

5.4 Translating the questionnaire

Finding the right words and phrases to translate the questionnaire can be difficult. Ideally, you want someone on your team who is highly proficient in English and the target language(s). Having a team member who is knowledgeable about specific innovation terms in various languages is also advantageous. If you are working with an external translator, be aware that they might lack specific subject knowledge. Since high-quality professional translations at native-speaker level are time-consum- ing to produce, be sure to set aside enough time and resources for this task. Use the OECD survey on private sector innovation as a resource for translating specific terms if it has been conducted in your country.

Testing the translated questionnaire(s) allows you to monitor whether the new phrasing is meaningful to respondents, but it will not tell you how accurate the translation is in the first place. It will not always be possible to provide a translation that is both accurate compared to the original questionnaire and meaningful to the respondents. If that is the case, prioritise the reactions of respondents and exercise caution when using the questions for international comparisons.

When translating the questionnaire, it is vital to use consistent terminology. For instance, do not use "private businesses" and "private enterprises" interchangeably; decide on one term and use it consistently. This eliminates a source of confusion for respondents and makes analysing and communicating the results easier and more accurate.

ACTIONABLE ADVICE 5.5
BEAR IN MIND THAT MULTILINGUAL QUESTIONNAIRES REQUIRE EXTRA WORK

Countries with more than one official language might require that you translate the questionnaire into multiple languages. It will be easier for respondents to answer the questionnaire if they can use their first language. Most modern survey software has a built-in option for creating multilingual surveys that allow respondents to select their language of choice. Your initial contact with respondents can be in the language most commonly used in their area or in all the languages they might prefer.

When conducting a multilingual survey, it is vital that questions are as similar as possible in each language to avoid the risk that the answers given depend on the language used.

Translating the questionnaire involves not only finding the right terminology in another language, but also ensuring that the questions and answer categories make sense in your national context. For instance, do not ask about collaboration with regional workplaces if your country does not operate at regional level. There may also be differences in the number of employees, for example, or the size of the average municipality, population and/or area to be considered. Again, try to find pragmatic solutions that fit the respondents’ real world. Be sure to also take cultural norms into consideration when determining how formally or informally you address respondents.