Sami Nenno on the typical problems of the most common service providers for conducting online surveys and how to find the right one for your own project.

Conducting surveys and designing questionnaires is common research practice, and not only for the social sciences. One might think that there is a vast array of good software tools out there. The first half is correct: when looking for software for setting up a survey, one is confronted with plenty of options. However, depending on what you want to do, it can be tricky to find a good tool that checks all the boxes. In the following sections, I will share some insights and experiences on how to find the right survey tool. I’ll describe some of the recurring problems one is confronted with and explain how one can identify different types of software which meet different requirements.

But first some words about the context of my experiences. I am a member of the research program Public Interest AI. As the name indicates, we are interested in the application of Artificial Intelligence (AI) in the public interest. We work from the premise that while there is plenty of reference to projects that claim to use AI for the public interest, there is little detailed information about how they do it. What types of AI or machine learning do they deploy? Are they publicly or privately funded? To what values and ethical guidelines do they subscribe? 

In order to find out, we decided to conduct a survey. Besides collecting data that might result in the empirical foundations of a research paper, our aim is to create an interactive map that displays Public Interest AI projects around the world. I am stressing this point because our approach comes with special demands. For our project, we needed a tool that would keep the survey online for several years and which automatically and in real time transfers the data to our website, so that the map is continuously updated. If you need your survey to be online for a shorter period of time and if you only need to access the data once, at the end of the field phase, you might have different expectations concerning the technical functionalities of the tool. However, besides this, our demands were quite in line with normal research practice.

Trial and Error

We started out with a classic tool that is extensively used in social sciences – LimeSurvey. LimeSurvey is a German product and this comes with the advantage that their servers are located in the European Union (EU). This makes life much easier, at least for research in EU member states, with regard to the strict standards on data security which apply in the Union. However, questionnaires created with LimeSurvey are difficult to embed in your own website. Furthermore, there is no elegant way to automatically transfer the data. Call this a technical problem. I will go into more detail about this below but first let me identify some more problem types:

After the first set-back, we tried out tools like AirTable, Google Forms, and Typeform. While these tools are easy to set up and easy to embed on your website, they seem to be designed for companies looking for user or customer feedback. As a result, they are better suited for shorter surveys that don’t come with ~20+ questions. Typeform, for instance, only displays one question per page. And the other two display the entire questionnaire on one page. Call this a design problem.

Our next choice was SurveyMonkey. Like Google Forms it is also easy to embed on your website and  – although more difficult – you can automatically transfer the data or add it to a GoogleSheet. SurveyMonkey’s advantage is that it presents fewer design problems. However, a major concern is that SurveyMonkey still refers to the EU-US Privacy Shield in their privacy policy, even though the European Court of Justice has ruled that the EU-US Privacy Shield is invalid. Call this a legal problem.

Motivated by the legal problem, we also thought of using tools with which you can create a survey from scratch. By this I mean that you do the programming and hosting yourself. The idea was that if we build a survey ourselves, there would be no legal problem regarding third parties. Vue Formulate is a nice way to do this. It is a library for Vue.js that is particularly designed for creating surveys. However, this brings us to the next and final type of problem – which you might have already guessed – is the problem of expertise. Do you know how to work with an IDE like Visual Studio Code? Do you know how to set up a Vue project? Or how to find a good host and take care of data protection yourself?

Four Types of Problems

Those are the most common problems one runs into when looking for a survey tool: technical, design, legal, and expertise. So what do they look like in detail?

1. Design Problems
By design problems I mean problems regarding the layout of the survey. This is not just a matter of aesthetics but also about functionality. If it is only possible to display one question at  a time, or all of them are listed on one page , respondents might get annoyed or intimidated and decide to quit the survey mid-way. In the worst case scenario, the tool only saves finished surveys, in the best case scenario it saves at least what the respondent already filled in (like Tally). However, even this survey might still be considered useless because large shares of unfinished surveys will skew your results. 

Another design problem is a limited choice of response scales. Depending on the survey, you will need different scales. A professional survey tool should provide (almost) all. One recurring problem is the text box for free text questions. Ideally, you can choose the size of your box yourself, so that you can suggest how much the respondent should write. Some tools, however, only allow for one-line answer boxes. And even if they do not have a limit on the amount of characters, they suggest a short answer.

2. Technical Problems
By technical problems, I mean problems regarding the functionality of the survey tool. As mentioned before, for us it was important that the data be automatically transferred to our website. The best way for doing this is via an API or by using Webhooks. However, many tools do not provide this functionality. Another way is to automatically load the data to Google Sheets. Some tools, like AirTable or Tally have an in-built function for this, others, like SurveyMonkey, allow for it by using a third-party software like Zapier. Note that relying on additional software like Google Sheets or Zapier might also come with privacy issues. 

Even if automatic data transfer is not one of your concerns, you might still want to embed the survey on your website, instead of (or in addition to) sending it via mail to the respondents. Luckily, this is a functionality that most tools provide. However, some tools do this in a less intuitive way than others. LimeSurvey , for example, offers the option but making it work takes a while and sometimes embedding the survey messes with its layout.

3. Legal Problems
Often you will collect personal information with your survey. The first question regarding this matter is: What personal data do I need for my research question and for how long do I need it? Depending on the answer, you should look for survey tools that meet those needs. Tally, for instance, only saves the response date, while LimeSurvey stores more metadata, like the respondent’s IP address. Sometimes, you need the email address to contact people for a follow-up survey, therefore asking for some personal data is unavoidable. However, a good survey tool should provide the option to separate the metadata from the responses in order to delete the former and provide anonymity for the latter. A good tool should also make it transparent to all users as to how long they will store the data. If they do not specify this or if they store it indefinitely, this certainly is a problem. 

Sometimes you cannot guarantee that you don’t receive personal information, even if you do not actively seek to. For instance, when your survey requires the respondent to write free texts. This is one of the reasons why checking the privacy guidelines of the respective tool is crucial. These, however, can be difficult to assess for people without legal expertise. Sometimes the company’s privacy policy is confusing because it is spread over multiple pages without clear order or there may be a mix of highly detailed explanations and very general statements. Both can be indicators that the company’s privacy policy is not the best. Unfortunately, recognizing those indicators requires some experience, because policy texts often are long and technical. 

As a rule of thumb – although this is not a substitute for consulting someone who is proficient in data protection law – one should check whether the tool requires an active acceptance to their data processing agreement (DPA). DPAs are a contract between you and the survey provider regarding the survey provider’s processing of personal data on your behalf and for your survey. Accepting the terms usually happens when signing up for an online account. Active acceptance does not necessarily involve signing something on paper, but you must have been directed to their terms and have the option to agree. The DPA should explicitly declare the company behind the tool as the data processor. Being the data processor, as opposed to being the data controller, means that they are not allowed to have an interest of their own. This must then be checked. As processors they cannot sell the data, use it for advertising purposes, or even for improving the tool. The only exception are IT security measures such as taking backups of the data.Sometimes, as in the case of Tally, the DPA is not linked from their website but requires you to ask for it. It then needs to be signed by both parties.

4. Problems of Expertise
The problem of expertise applies almost exclusively to tools for setting up a survey from scratch. Such tools are libraries, i.e. extensions to Javascript frameworks like Vue.js or React. It is difficult to say what level of programming expertise is required. You don’t have to be a professional or experienced (web) developer, that’s for sure. However, if you have no experience with HTML, CSS, and Javascript (Frameworks) or with programming languages like Java, Python, and so on, you might be better off with higher level tools like Tally or Alchemer. 

There are some things you have to be prepared for when creating a survey from scratch – if you feel confident with them, a tool like Vue Formulate might be the right choice for you, but requires some extra steps. First, you have to find a web host. It should be reliable, not too expensive, have good user support, take regular backups, and have the servers located in the EU. This is crucial, because otherwise you will end up with data security issues, which is one one of the problems you tried to avoid in the first place. Second, once you have uploaded your own script to the servers, you have to take care of data protection. This includes (but is not limited to) making sure that the access details to the database are not visible in the plaintext and this requires you to have expertise in basic communication encryption.

Three Types of Survey Tools

When searching through the jungle of survey tools, we loosely identified three types or clusters of tools that share a family resemblance. Those categories are partly based on the problem types. However, they are also based on soft criteria, like the appearance of the website and what use-cases the respective company has in mind.

1. Business tools
By calling them business tools, I don’t want to imply that they cannot be used for scientific purposes, too. However, it seems to me that those tools, for example AirTable, Typeform or Google Forms, are rather built for business purposes like user feedback or contact forms. They all have the aforementioned design problems in common. 

Despite their original purpose, those tools can be a good fit for (very) short surveys. They are fairly intuitive and easy to share. AirTable is among the best in terms of data transfer, because it provides its own API. And for Google Forms, you need nothing but a Google account. Therefore, regarding technical functionality and levels of expertise, they are great choices. But once the survey contains many questions, business tools are not well equipped for the job.

2. Scientific tools
By scientific tools, I refer to those tools which come with all or almost all the design features needed to create a professional scientific survey. Examples are Alchemer, Qualtrics, SurveyMonkey, LimeSurvey, EU Survey, and Tally. With those tools it is possible to design a survey that should meet all professional requirements. 

However, despite their lack of design problems, scientific tools can differ a great deal. First, with respect to their price. EU Survey was developed in the course of the ISA² Programme and is a free tool. Qualtrics on the other hand, can cost more than U$1000/year and similarly,  tools like SurveyMonkey or Alchemer come close to that. Regarding transparency and data security EU Survey is a good choice, since it is open-source and maintained by an EU-project. SurveyMonkey on the other side, processes and uses the answers they receive and sells insights they draw from them. Finally, tools like Alchemer or Tally make it easy to embed your survey on your own website and automatically transfer the data, while for EU Survey and LimeSurvey this is difficult.

3. Technical tools
Tools of this category require technical skills, like experience with frontend and backend engineering. Examples are libraries for Vue.js like Vue Formulate or VueJS Generators and libraries for React like Formik. As described, those tools only make sense for people with the required skills. However, their major advantage is that you have free choice regarding many topics. Most importantly, they make your life easier with regards to data protection because they do not require a third party between you and the respondent.

And the winner is…

The tool we used in the end was Tally. Tally is a fairly young company that launched at the end of 2020. Given that there are a lot of survey tool providers on the market, it may have been a risky choice. However, Tally checked all the boxes. They are based in Belgium, so we could expect that data privacy laws were enforced according to European standards. Don’t be surprised by their domain (.so),  it is  a popular domain among tech companies. Tally’s (small) team is adding functionality on an almost weekly basis and is open to suggestions. They also have good user support, including a Slack channel, and they usually don’t take longer than a day to reply. Unlike Alchemer, which also checks all boxes, Tally is free (although there is a paid version, too). I am not mentioning this because we are stingy. Often tools and other infrastructures that are set up during a research project are not maintained once the funding runs out. This is a pity because it could be used by other projects or organisations. Free tools have the potential of filling this gap because they are easier to be taken over by others.

Summing up, I have created  a table of the tools we encountered during our search. The table contains information on thirteen tools with respect to the four problem types. There is also a column with a final score based on how many boxes the tool checks. Note that a high score does not necessarily mean that this is the best tool. If a tool checks all but one box because it has an insufficient data security policy, it is perhaps a no-no.