Part 4: How to find respondents – and reach them

Part 4 is designed to help:

→ Define the public sector in your country

→ Find your way to a data source on public sector workplaces

→ Draw a sample of workplaces

→ Make a strategy for reaching a relevant person within each workplace

→ Maximise your response rate

→ Assess whether the survey is representative

 

Table of contents:

4.1 Identifying respondents

4.2 Step 1: Define the public sector in your country

4.3 Step 2: Find a data source

4.4 Step 3: Define the survey population

4.5 Step 4: Contact your respondents

4.6 Consider establishing a panel of respondents for repeating the survey

4.7 Step 5: Are the answers representative?

4.8 An overview

Comments

 

If you're reading this, you are most likely a pioneer. You are considering gaining new ground by conducting the first truly representative national survey of public sector innovation in your country. As a forerunner, expect to encounter strenuous and surprising challenges on your path to new horizons. Also, expect to make some mistakes – or at least not to get a perfect first result. That is the essence of pioneering work.

Fortunately, after reading this part, your surprises will be fewer as your backpack will be full of knowledge on the many challenges other pioneers have encountered.

4.1 Identifying respondents

Finding and reaching your respondents – the people who are going to provide the data the Innovation Barometer is built on – is possibly the hardest and most time- consuming part of measuring public sector innovation. Overall, finding respondents is a tradeoff between resources and data quality. The more resources available, the more you can ensure a higher response rate and more representative data, increasing the data quality and the legitimacy of the survey overall. However, if resources are limited you might have to make do with a lower response rate and less knowledge about who responded to the survey and who did not.

Fewer resources

More resources

Low response rate
+ non-representative data
= lower data quality and legitimacy

High response rate
+ representative data
= high data quality and legitimacy


Ideally you want:

  1. A sample of public sector workplaces that represents all public sector workplaces (the population) as closely as possible
  2. One person within as many sample workplaces as possible to answer the survey (a high response rate)
  3. The answers to be representative of all public sector workplaces

The essential question left to answer is how to maximise these steps with the resources and options available to you.

The logic of why each step is important is better understood working backwards, as shown in figure 4.1 below. The answers given in the survey must be representative of all public sector workplaces so that when you use the statistics, you are not just representing workplaces that answered the survey, but the public sector as a whole. That is, you want to assume that if the responses show that eight out of ten workplaces have introduced an innovation, this share is also true for those public sector workplaces not directly represented in the survey.

Critics will also have a harder time dismissing your results if your methodology is solid.

Figure 4.1: Depiction of the logic of moving from a population of public sector workplaces to respondents and back again. A sample of workplaces is drawn from the population and the survey is sent to one person (the manager) within each workplace. Only some of them will fill out the survey, the respondents. To ensure representation you need to know to what extent each step represents the former.

ACTIONABLE ADVICE 4.1 
ESTABLISH YOUR TEAM

If you are not experienced in survey methodology, collaborate with someone who is. You will need their specialised knowledge to transform the information in the following pages into an actual data collection process. Typically, people with this knowledge can be found in your national statistical office, research institutions, private consultancies and research/analysis units in larger public organisations. The titles might include analysis, data, research or development and they often have a background in social sciences with a specialisation in quantitative research.

4.2 Step 1: Define the public sector in your country

Although you probably have an intuitive understanding of the public sector in your own country, you need to make the definition of the public sector clear to yourself and others. As described in part 3.5, public sector workplaces are not always easy to define, and the same holds true for the public sector in general. Defining the public sector early on will help you know what to look for when searching for a data source and making practical decisions on what to include or not. In Part 5 on adapting the questionnaire, we advise you to test in your national context. These tests can also provide practical corrections to your public sector definition. If a respondent disagrees about being defined as part of the public sector, the lines may need to be drawn elsewhere.

A clear definition of the public sector is crucial when doing international comparisons. The size and functions of the public sectors in different countries vary greatly and being aware of these variations is the first step in making meaningful compari- sons of countries where the public sectors differ.

USE CASE
DEFINING THE PUBLIC SECTOR

In the pilot for the German Innovationskompass the public sector was defined as the federal government (including special foundations), state governments, municipalities and associations of municipalities (including their facilities such as hospitals), and the budgets of social insurance institutions, e.g. statutory health insurance funds or pension insurance institutions.

4.3 Step 2: Find a data source

Three possible options for finding your data source are described below.

ACTIONABLE ADVICE 4.2 
BE TRANSPARENT

Carefully describe what you have done in every step so that others know what reservations to make when using your data or making comparisons between countries. The data sources you rely on are not always similar, so transparency is essential.

Option A: Collaborate with your national statistical office

To find potential respondents in the first place, you need a data source on public sector workplaces. That data source should ideally represent the full population of public sector workplaces to ensure representative data. If your data source includes additional information about the workplaces, for example number of employees, type of activity or location area, these background variables can be used to weight the final responses to represent the population more accurately in terms of those variables.

Your national statistical office is most often the natural starting point for finding respondents and reaching respondents. They are likely to have a registry of public sector workplaces and will have the knowledge on how to conduct a survey. Collaborating with your national statistical office is also a way to legitimise your survey.

"Many countries, including in the EU, are obliged to maintain a business register covering both business and public administration. These business registers are used for many purposes, e.g. as a base for drawing survey samples, and can be an important source for finding your respondents. Other sources may also be available in your statistical office, e.g. registers on educational institutions, local administration units and public health institutions , or statistics on public finances or national accounts. So, start by taking a look at what statistics are available online from your national statistical office that cover the parts of the public sector that you are interested in for your survey."

Helle Månsson, Research, Senior Adviser, Technology and Culture, Business Statistics
Statistics Denmark

Option B: Using another central data source

If collaborating with your national statistical office is not an option, put in sufficient time and resources to research if anyone else has an existing data source on public sector workplaces that you can use. Governmental offices or organisations for local government are likely to have some usable (member) databases, and you can also consider universities or other research institutions. You can also research what  data sources other surveys on public sector workplaces have used or are currently using. Gaining access to a data source is another great opportunity for strategic collaboration, as described in part 1.

ACTIONABLE ADVICE 4.3
SET ASIDE TIME AND RESOURCES TO IDENTIFY THE ACTORS WHO CAN LEAD YOU TO A DATA SOURCE

You do not have to cover the public sector in its entirety – the population can be defined as a certain part of the public sector or as a single organisational level (as for instance governmental offices only). Make sure to be transparent about what sectors, branches and types of activities are included in the survey.

  • Your national statistical office
  • Organisations for local or regional government
  • Central government offices
  • Universities and research institutions 
  • Private research agencies

USE CASE 
USING SELF-REPORTED LOCAL AND REGIONAL GOVERNMENT MEMBER INFORMATION 

To obtain a list of local and regional level workplaces for the Norwegian Innovation Barometer, the Norwegian Association of Local and Regional Authorities called on its own publisher, which specialises in maintaining, proofing and supplying data on municipalities and regions. However, its data rely on self-reported contact information on managers and the information is not always complete or up to date. Various registries and data sources, including municipal and regional websites, were used to supplement the original lists.

The data quality of the lists varies greatly from one policy area to another. In education and upbringing, the lists were almost complete, just as the quality was also quite high in health and care. Obtaining contact information on managers within the social services proved difficult as it is publicly unavailable. Consequently, the Association had to rely on the central administration of social services to forward the survey to the relevant managers.

Option C: Rely on administration

If you are unable to find a useful existing data source on public sector workplaces, consider collaborating with various organisations such as municipalities, that can provide data on their workplaces. With a good research design, you can use informa- tion from these organisations to present statistics on the public sector in general.

You need to give detailed instructions to the organisations on what constitutes a workplace to make sure they are defined in the same way in different organisations (more on this in part 3).

It is preferable if the organisations will let your organisation contact the respondents yourself, as this will make managing the data collection process easier. Another option is to rely on administrative units to forward the survey to public sector workplaces as instructed. This makes data collection harder as non-response might occur on two levels: the workplace fails to answer the survey and the administration fails to forward the survey.

Make sure someone on your team knows the methodological consequences of two-tier sampling, for example in relation to weighting data and statistical uncertainty in working with a limited number of overall units.

4.4 Step 3: Define the survey population

After identifying your population, you can go two ways: either carry out a survey for the total population or for a smaller sample. If your population is small, e.g. less than 500 workplaces, it is advisable to include all of them. If your population is large, using a sample means fewer people have to spend time answering the survey and fewer resources will be needed to contact respondents. Statistical representation of your population can be achieved by drawing the sample randomly from your data source.

If your population is believed to be homogeneous for certain parameters (e.g. type of activity) you can still draw a sample, assuming that it will be representative of the rest of the population. This may be refined further by grouping the population into strata: If background data, e.g. number of employees are available, the population can be stratified into size classes (e.g. less than 10 employees, 10–19 employees, 20–49 employees) and then samples can be drawn from each stratum. This ensures that the survey population consists of workplaces in all the different size classes.

If other variables like the type of activity, e.g. nurseries, elderly care, hospitals and public administration is available further refinement is possible, allowing you to cross size-class by type of activity and draw the sample from each of these strata.

Another option is to oversample or sample all workplaces in some groups/strata to make sure you get enough answers for groups of special interest. Depending on your strategy and communication plan, this may involve groups where you plan to do some separate analysis to communicate targeted results. For non-random sampling, make sure to weight your final data to correct the skewness created.

A relatively large sample compared to the population provides more statistical certainty, allowing a more detailed statistical analysis, depending of course on the response rate. But the survey also becomes more expensive. If mostly interested in the overall distributions of each question, you can manage with a relatively smaller sample, although this decision should be made with someone knowledgeable about surveys to make sure the sample fits your needs. To sum up, sample size is important but knowing who is in the sample is more important.

ACTIONABLE ADVICE 4.4
SPEND YOUR RESOURCES WISELY

Concentrating your resources on gaining a high response rate in a small sample ensures better data quality than working with a larger sample and obtaining a lower response rate, even if the final number of responses is similar.

USE CASE 
COLLABORATING WITH THE NATIONAL STATISTICS OFFICE 

The population data source used in the Danish Innovation Barometer is Statistics Denmark’s database on public and private sector workplaces (excluding military workplaces). The public sector workplace registry includes about 15,000 workplaces with at least three employees (including manager). It also includes variables about each workplace, such as type of activity, the name and address of each workplace and the number of employees.

The workplaces are defined by geographical location, which means that large entities like hospitals or administrative centres are registered as one workplace, though they might realistically function as several workplaces sharing one address. The registry relies on, for instance municipalities to self-report when workplaces are established, modified or closed down. Consequently, data are not always up to date.

Even though the registry was a good starting point, decisions had to be made about how to handle out-of-date information but also workplaces that did not fit into the categories for types of activity or that had employees with no physical workplace.

Using type of activity and workplace size (number of employees), an initial sample comprising about 7,000 public workplaces was created, with deliberate oversampling of smaller workplaces and workplaces in types of activities with few workplaces to ensure an adequate absolute number of workplaces in these groups (stratification). The sample size was quite large to enable detailed analysis within subgroups, e.g. for specific types of activities.

USE CASE 
INNOVATION CAPACITY IN PUBLIC SECTOR ORGANISATIONS

The Department of Innovation and Best Practices of the Greek Ministry of Interior conducted a survey on innovation capacity in public sector organisations that implement governmental policy, collecting data at three levels of governance: Central, regional and local.

A population of 700 organisations was selected as they implement core governmental policies, whereas a large number of smaller organisations were excluded from the initial population. Data on the 700 organisations were sourced from the national repository for the public sector, which is operated by the Ministry of Interior.

The questionnaire, designed to be applicable for different types of organisations, was sent to higher-level managers, with an invitation by the Minister of Interior to complete it. Many of the organisations that did not initially reply had a common trait, they were members of the innovation network facilitated by the Ministry of Interior. After network representatives were contacted personally, responses from higher-level managers increased.

www.innovation.gov.gr

4.5 Step 4: Contact your respondents

To enable a workplace to participate in the survey, you need to know who to contact. This may sound easy, but do not underestimate how time-consuming finding and validating contact information is, even if you are lucky enough to have a data source that contains contact information.


WARNING!

Do not underestimate the task of finding and correcting contact information for managers for each workplace in the sample. Be sure to allocate enough time and resources for this step as it can take longer than anticipated.

The Innovation Barometers in the Nordic countries exclusively used web-based surveys to collect data and sent invitations by email. However, the approach described here can be modified to fit other methods, including invitations by letter.

The ideal person for answering the survey within a workplace is the one who knows the most about its innovation activity. If you do not know who that is, the second-best person to contact is the manager, who either likely knows enough to complete the survey or knows who can. Including a question about who answered the survey makes it possible to analyse whether survey results differ when forwarded to some- one besides the manager.

To find contact information use registries when available, search online or contact workplace administrative units.

ACTIONABLE ADVICE 4.5
CONTACT PEOPLE DIRECTLY

Contact the manager of each workplace directly via their personal email address (firstname.lastname@fictitiousmunicipality.org). Contacting a specific person clearly defines who is responsible to act on the survey and works better than writing to a generic or administrative email address (info@fictitiousmunicipality.org).

USE CASE
OUTSOURCING THE TASK OF FINDING CONTACT INFORMATION TO AN EXPERT RESEARCH AGENCY

In 2018 the Norwegian Association of Local and Regional Authorities hired a private research agency, Ipsos, to find contact information on local and regional workplaces in the five policy areas the Norwegian Innovation Barometer covers. In the 2020 data collection even more time was spent on this task, with about 40 person days devoted to providing good-quality lists of physical addresses and relevant email addresses. This ensured representation in all policy areas in the survey for all municipalities and regions and also in diverse types of workplaces.

Obtaining direct contact information for managers has become harder since the introduction of the European General Data Protection Regulation, which protects people’s personal data. The only way to reach some managers involved writing to a central administrative email address and relying on them to forward the emails.

To make the task of forwarding emails more feasible for central administrations, a cap on the number of emails sent to the same central email address was imposed and varying according to the size of each municipality or region. As a result, some groups are underrepresented in the survey compared to the original data source used.

ACTIONABLE ADVICE 4.6
TAKE PRECAUTIONS WHEN WORKING WITH PRIVATE CONSULTANCIES

  • If collaborating with a private consultancy on data collection, it is important to match expectations. Finding information on respondents takes tremendous work, and the hours needed can be difficult to estimate. Take into account this uncertainty when entering into a contract.
  • Quality check each step of the consultancy’s work, even if you have worked with them before, especially if the task is proving to be difficult.
  • Be aware of national and international regulations of how private consultancies process respondents’ personal information.

The email invitation can be further personalised to include the names of the manager and the workplace. This will improve response rates. The invitation's wording can also be adapted to match the specific workplace. Including the name of the work- place helps the manager to focus when answering the survey and avoids confusion when the boundaries definition of the specific workplace do not completely match the manager’s perspective.

ACTIONABLE ADVICE 4.7
PERSONALISE CONTACT

If you have the information – or are willing to spend resources on finding it – personalise the email invitation by:

  • Including the manager’s name
  • Including the name of the workplace Including number of employees in the workplace
  • Customising wording to specific types of workplaces

An example is shown in Appendix 3. Email invitation sent to respondents who are asked to answer the Questionaire.

USE CASE
PERSONALISING EMAIL INVITES INCREASES RESPONSE RATE

Personalising email invitations might not sound important, but it helps increase the response rate. In the first round of the Danish Innovation Barometer, email invitations were sent to administrative email accounts when no personal email address could be found. This produced a response rate of 33%. In the second round the response rate improved to 50% by personalising the email invitation and limiting the sample to workplaces that could be directly contacted.

A private research institute was hired to look up information on name, professional email address and phone number for the manager of each workplace in the sample and to check if the name of the workplace was correct. About 2,000 workplaces were removed from the sample in this process as they no longer existed or the contact information was impossible to find.

People contacted to take part in the survey were addressed by name and their workplace was included in the invitation. Two email reminders were sent. For sectors and types of workplace activities with a low response rate, some respondents were also contacted by phone. The respondents were encouraged to forward the invitation email/reminders to someone else within the workplace if another person knew more about the workplace’s innovation activities.

Fewer resources

More resources

info@municipality.org
Dear manager at unspecified workplace

first.last@municipality.org
Dear Firstname Lastname at specific workplace

 

The response rate can be influenced by the impression you create as the sender. When people know the organisation behind the survey and trust that the results will be used constructively and respectfully, they are more likely to respond. A good communication strategy can also help increase the response rate. If people know the survey is coming they will be more likely to answer it.

Ideally, each respondent should be invited to participate via a personal response link. This allows you to see exactly who has replied and to send email reminders exclusively to non-respondents. Knowing who has answered the survey is also necessary for analysing whether answers are representative.

Respondents can be phoned after a few email reminders, but this can be time-consuming and expensive. However, you could decide to strategically boost the response rate in groups with a low response rate by phone contact.


WARNING!

If you do not send your respondents a personal response link or key, you will not be able to send them reminders to answer the survey. Also, you will not know exactly who replied when the data collection is complete

ACTIONABLE ADVICE 4.8
DO NOT HESITATE TO SEND REMINDERS

Do not be afraid to use email or phone reminders to encourage people to answer the survey. Surprisingly, many respondents will forget or ignore the first email invitation but will respond to the second or third.

4.6 Consider establishing a panel of respondents for repeating the survey

If you are planning to repeat the survey, consider having a panel of respondents, which has several advantages. The time involved in finding and reaching respond- ents will be reduced as you have their contact information. You will have identified  a part of your population and know their characteristics. When respondents answer the survey for a second or subsequent time, they will be familiar with definitions  and know where to find the data in the organisation, which increases the likelihood of them responding.

When retaining contact information in the European Union to reuse in a subsequent survey, the respondent must be in agreement in order to comply with the European General Data Protection Regulation. It is important to state that the respondent will only be contacted for a new public innovation survey. It is also important to state how long the contact information will be kept and when the next survey will be sent. In general, the more relevant information you can provide the respondent the better. Also, think about how the respondent can be motivated, e.g. by sending them the survey results and inviting them to (digital) seminars on the outcomes.

Having a panel will not stop you from having to find new respondents, for example, if people leave their current position. However, the main benefits of maintaining a panel are that you will save time and have more experienced respondents.

4.7 Step 5: Are the answers representative?

When the data collection is complete, statistical analysis will show if the basic characteristics of responding workplaces match the population of workplaces. When this is the case, then the survey is representative of the population. To do this analysis, you need some background data on the full population to compare with the respondents, (e.g., workplace size, location or type of activity), which could be in the original data source or compiled when looking for contact information. Using answers only from the survey is not a solution, as the data for non-respondents will not be available.

ACTIONABLEE 4.9 
MAKE WEIGHTING DATA AN OPTION

Decide early on if you want to be able to weight the data. You can then decide to find or fact check variables for weighting while searching for managers’ contact information.

Variables for weighting data: 

  • Workplace size
  • Sector
  • Geographical location
  • Service area (type of activity)

If the answers do not match the population, by accident or by deliberate sampling, then answers from underrepresented groups can receive a higher weight, and overrepresented groups can receive a lower weight when doing the analysis. The same variables used for checking if the responses are representative can be used for weighting the answers.

If you end up with a non-representative Innovation Barometer, do not despair. You will still have better insight into public sector innovation than you had before.


WARNING!

Weighting data is a way to compensate for entities that did not respond to the survey, making it possible to analyse the total population, e.g. to be able to calculate the percentage of innovative workplaces in the population. You will never know if there is something you did not measure which separates the respondents from the non-respondents. For example, innovative workplaces may be more likely to reply to a survey on innovation. Some methods can shed light on this issue, for example, comparing replies from those who answered promptly to those who replied only after several reminders. Doing a non-response survey is another way to find out if there are differences between the entities that responded and those that did not.

The respondents might also choose to answer the survey in a way that sheds a positive light on their own workplace. These basic issues are inherent in survey science and are in no way unique to surveys on innovation.

4.8 An overview

You need …

Ideally you should ...

If your resources or options are limited ...

A data source on public sector workplaces (a sample) that represents all public sector workplaces (the population) as closely as possible; the number of workplaces in the sample is not as important as whether they represent the population as a whole

- Collaborate with the national statistical office

- Put in the time and resources, mixed with a bit of luck, to find a great data source

- Use a subset of the public sector

- Use a unit other than workplaces- Accept to not know much about the population

To invite every workplace in the sample to be part of the survey and for as many workplaces from the sample as possible to answer the survey (high response rate)

- Find name and contact information on someone who can answer questions about the workplace’s innovation activity

-Appear a trustworthy sender

- Personalise invitations

- Send email reminders

- Use phone reminders or phone interviews as a follow-up to non-respondents

- Use a subset of the public sector

- Use a unit other than workplaces

- Accept to not know much about the population

The answers to be representative of the population

- Weight data by known background knowledge, e.g. workplace size, location and type of activity

- Use phone reminders or phone interviews strategically as a follow-up to underrepresented non-respondents

- Do not claim that the answers represent the population

 

Download part 4

Download a full PDF-version of part 4: How to find respondents – and reach them

Download part 4 (PDF)

Comments

Paul Sauer (COI - Denmark)

26. januar 2021 - 11:26

Please write your own comment below.

Write your own comment