Sami Nenno on the typical problems of the most common service providers for conducting online surveys and how to find the right one for your own project.
Conducting surveys and designing questionnaires is common research practice, and not only for the social sciences. One might think that there is a vast array of good software tools out there. The first half is correct: when looking for software for setting up a survey, one is confronted with plenty of options. However, depending on what you want to do, it can be tricky to find a good tool that checks all the boxes. In the following sections, I will share some insights and experiences on how to find the right survey tool. I’ll describe some of the recurring problems one is confronted with and explain how one can identify different types of software which meet different requirements.
But first some words about the context of my experiences. I am a member of the research program Public Interest AI. As the name indicates, we are interested in the application of Artificial Intelligence (AI) in the public interest. We work from the premise that while there is plenty of reference to projects that claim to use AI for the public interest, there is little detailed information about how they do it. What types of AI or machine learning do they deploy? Are they publicly or privately funded? To what values and ethical guidelines do they subscribe?
In order to find out, we decided to conduct a survey. Besides collecting data that might result in the empirical foundations of a research paper, our aim is to create an interactive map that displays Public Interest AI projects around the world. I am stressing this point because our approach comes with special demands. For our project, we needed a tool that would keep the survey online for several years and which automatically and in real time transfers the data to our website, so that the map is continuously updated. If you need your survey to be online for a shorter period of time and if you only need to access the data once, at the end of the field phase, you might have different expectations concerning the technical functionalities of the tool. However, besides this, our demands were quite in line with normal research practice.
Trial and Error
We started out with a classic tool that is extensively used in social sciences – LimeSurvey. LimeSurvey is a German product and this comes with the advantage that their servers are located in the European Union (EU). This makes life much easier, at least for research in EU member states, with regard to the strict standards on data security which apply in the Union. However, questionnaires created with LimeSurvey are difficult to embed in your own website. Furthermore, there is no elegant way to automatically transfer the data. Call this a technical problem. I will go into more detail about this below but first let me identify some more problem types:
After the first set-back, we tried out tools like AirTable, Google Forms, and Typeform. While these tools are easy to set up and easy to embed on your website, they seem to be designed for companies looking for user or customer feedback. As a result, they are better suited for shorter surveys that don’t come with ~20+ questions. Typeform, for instance, only displays one question per page. And the other two display the entire questionnaire on one page. Call this a design problem.
Motivated by the legal problem, we also thought of using tools with which you can create a survey from scratch. By this I mean that you do the programming and hosting yourself. The idea was that if we build a survey ourselves, there would be no legal problem regarding third parties. Vue Formulate is a nice way to do this. It is a library for Vue.js that is particularly designed for creating surveys. However, this brings us to the next and final type of problem – which you might have already guessed – is the problem of expertise. Do you know how to work with an IDE like Visual Studio Code? Do you know how to set up a Vue project? Or how to find a good host and take care of data protection yourself?
Four Types of Problems
Those are the most common problems one runs into when looking for a survey tool: technical, design, legal, and expertise. So what do they look like in detail?
1. Design Problems
By design problems I mean problems regarding the layout of the survey. This is not just a matter of aesthetics but also about functionality. If it is only possible to display one question at a time, or all of them are listed on one page , respondents might get annoyed or intimidated and decide to quit the survey mid-way. In the worst case scenario, the tool only saves finished surveys, in the best case scenario it saves at least what the respondent already filled in (like Tally). However, even this survey might still be considered useless because large shares of unfinished surveys will skew your results.
Another design problem is a limited choice of response scales. Depending on the survey, you will need different scales. A professional survey tool should provide (almost) all. One recurring problem is the text box for free text questions. Ideally, you can choose the size of your box yourself, so that you can suggest how much the respondent should write. Some tools, however, only allow for one-line answer boxes. And even if they do not have a limit on the amount of characters, they suggest a short answer.
2. Technical Problems
By technical problems, I mean problems regarding the functionality of the survey tool. As mentioned before, for us it was important that the data be automatically transferred to our website. The best way for doing this is via an API or by using Webhooks. However, many tools do not provide this functionality. Another way is to automatically load the data to Google Sheets. Some tools, like AirTable or Tally have an in-built function for this, others, like SurveyMonkey, allow for it by using a third-party software like Zapier. Note that relying on additional software like Google Sheets or Zapier might also come with privacy issues.
Even if automatic data transfer is not one of your concerns, you might still want to embed the survey on your website, instead of (or in addition to) sending it via mail to the respondents. Luckily, this is a functionality that most tools provide. However, some tools do this in a less intuitive way than others. LimeSurvey , for example, offers the option but making it work takes a while and sometimes embedding the survey messes with its layout.
3. Legal Problems
Often you will collect personal information with your survey. The first question regarding this matter is: What personal data do I need for my research question and for how long do I need it? Depending on the answer, you should look for survey tools that meet those needs. Tally, for instance, only saves the response date, while LimeSurvey stores more metadata, like the respondent’s IP address. Sometimes, you need the email address to contact people for a follow-up survey, therefore asking for some personal data is unavoidable. However, a good survey tool should provide the option to separate the metadata from the responses in order to delete the former and provide anonymity for the latter. A good tool should also make it transparent to all users as to how long they will store the data. If they do not specify this or if they store it indefinitely, this certainly is a problem.
As a rule of thumb – although this is not a substitute for consulting someone who is proficient in data protection law – one should check whether the tool requires an active acceptance to their data processing agreement (DPA). DPAs are a contract between you and the survey provider regarding the survey provider’s processing of personal data on your behalf and for your survey. Accepting the terms usually happens when signing up for an online account. Active acceptance does not necessarily involve signing something on paper, but you must have been directed to their terms and have the option to agree. The DPA should explicitly declare the company behind the tool as the data processor. Being the data processor, as opposed to being the data controller, means that they are not allowed to have an interest of their own. This must then be checked. As processors they cannot sell the data, use it for advertising purposes, or even for improving the tool. The only exception are IT security measures such as taking backups of the data.Sometimes, as in the case of Tally, the DPA is not linked from their website but requires you to ask for it. It then needs to be signed by both parties.
4. Problems of Expertise
There are some things you have to be prepared for when creating a survey from scratch – if you feel confident with them, a tool like Vue Formulate might be the right choice for you, but requires some extra steps. First, you have to find a web host. It should be reliable, not too expensive, have good user support, take regular backups, and have the servers located in the EU. This is crucial, because otherwise you will end up with data security issues, which is one one of the problems you tried to avoid in the first place. Second, once you have uploaded your own script to the servers, you have to take care of data protection. This includes (but is not limited to) making sure that the access details to the database are not visible in the plaintext and this requires you to have expertise in basic communication encryption.
Three Types of Survey Tools
When searching through the jungle of survey tools, we loosely identified three types or clusters of tools that share a family resemblance. Those categories are partly based on the problem types. However, they are also based on soft criteria, like the appearance of the website and what use-cases the respective company has in mind.
1. Business tools
By calling them business tools, I don’t want to imply that they cannot be used for scientific purposes, too. However, it seems to me that those tools, for example AirTable, Typeform or Google Forms, are rather built for business purposes like user feedback or contact forms. They all have the aforementioned design problems in common.
Despite their original purpose, those tools can be a good fit for (very) short surveys. They are fairly intuitive and easy to share. AirTable is among the best in terms of data transfer, because it provides its own API. And for Google Forms, you need nothing but a Google account. Therefore, regarding technical functionality and levels of expertise, they are great choices. But once the survey contains many questions, business tools are not well equipped for the job.
2. Scientific tools
By scientific tools, I refer to those tools which come with all or almost all the design features needed to create a professional scientific survey. Examples are Alchemer, Qualtrics, SurveyMonkey, LimeSurvey, EU Survey, and Tally. With those tools it is possible to design a survey that should meet all professional requirements.
However, despite their lack of design problems, scientific tools can differ a great deal. First, with respect to their price. EU Survey was developed in the course of the ISA² Programme and is a free tool. Qualtrics on the other hand, can cost more than U$1000/year and similarly, tools like SurveyMonkey or Alchemer come close to that. Regarding transparency and data security EU Survey is a good choice, since it is open-source and maintained by an EU-project. SurveyMonkey on the other side, processes and uses the answers they receive and sells insights they draw from them. Finally, tools like Alchemer or Tally make it easy to embed your survey on your own website and automatically transfer the data, while for EU Survey and LimeSurvey this is difficult.
3. Technical tools
Tools of this category require technical skills, like experience with frontend and backend engineering. Examples are libraries for Vue.js like Vue Formulate or VueJS Generators and libraries for React like Formik. As described, those tools only make sense for people with the required skills. However, their major advantage is that you have free choice regarding many topics. Most importantly, they make your life easier with regards to data protection because they do not require a third party between you and the respondent.
And the winner is…
The tool we used in the end was Tally. Tally is a fairly young company that launched at the end of 2020. Given that there are a lot of survey tool providers on the market, it may have been a risky choice. However, Tally checked all the boxes. They are based in Belgium, so we could expect that data privacy laws were enforced according to European standards. Don’t be surprised by their domain (.so), it is a popular domain among tech companies. Tally’s (small) team is adding functionality on an almost weekly basis and is open to suggestions. They also have good user support, including a Slack channel, and they usually don’t take longer than a day to reply. Unlike Alchemer, which also checks all boxes, Tally is free (although there is a paid version, too). I am not mentioning this because we are stingy. Often tools and other infrastructures that are set up during a research project are not maintained once the funding runs out. This is a pity because it could be used by other projects or organisations. Free tools have the potential of filling this gap because they are easier to be taken over by others.
Summing up, I have created a table of the tools we encountered during our search. The table contains information on thirteen tools with respect to the four problem types. There is also a column with a final score based on how many boxes the tool checks. Note that a high score does not necessarily mean that this is the best tool. If a tool checks all but one box because it has an insufficient data security policy, it is perhaps a no-no.
|Name||Design||Technical functionality||Data Security||Requires Expertise||Score|
|Air Table||Bad. Only one page for the entire survey.||Good. Easy to embed on your own website. Data can be automatically transferred to Google Sheets and others. Provides API, too.||Mixed. Uses data to “to provide, maintain, or improve the Services”. However, strictly speaking, they should not do the latter. Offers Data Processing Addendum that is signed by both parties.||Good. User friendly interface and easy to get started.||2.5/4|
|Typeform||Bad. One page per question.||Good. Easy to embed on your own website. Data can be automatically transferred to Google Sheets.||Good. Clear Data Processing Agreement, which declares them as the data processor. Moreover, as a spanish company, you can expect that the laws are enforced according to european standards.||Good. User friendly interface and easy to get started.||3/4|
|Google Form||Bad. Only one page for the entire survey.||Good. Easy to embed on your own website. Data can be automatically transferred to Google Sheets.||Mixed. Some institutional accounts come with a data processing agreement. But finding out for private accounts can be difficult.||Good. User friendly interface and easy to get started.||2.5/4|
|Tally||Good. Has (almost) all possible question types and offers multiple-pages surveys.||Good. Easy to embed on your own website. Data can be automatically transferred to Google Sheets and others. Provides Webhooks, API is planned.||Good. The information on their website is too sparse, so you have to individually ask for a data processing agreement that both parties have to sign. But they quickly provide you with that.||Good. User friendly interface and easy to get started.||4/4|
|LimeSurvey||Good. Has all possible question types and offers multiple-pages surveys.||Bad. Embedding is difficult and automatic data transfer is not provided.||Good. Additionally, their servers are located in the EU.||Good. User friendly interface and easy to get started.||3/4|
|Alchemer||Good. Has all possible question types and offers multiple-pages surveys.||Good. Easy to embed on your own website. Data transfer via Secure File Transfer Protocol (SFTP).||Good. No own use of data, conclusion of a commissioned data contract, explanation of the ECJ ruling on EU-US Privacy Shield and how they deal with it and what consequences it has and does not have.||Good. User friendly interface and easy to get started.||4/4|
|Jotform||Good. Has all possible question types and offers multiple-pages surveys.||Good. Easy to embed on your own website. Data can be automatically transferred to Google Sheets and others. Provides Webhooks.||Mixed. Does not sell the collected data and deletes it latest one month after account is deleted. Offers Data Processing Addendum that is signed by both parties. However, still refers to EU-US Privacy Shield.||Good. User friendly interface and easy to get started.||3.5/4|
|Qualtrics||Good. Has all possible question types and offers multiple-pages surveys.||Good. Easy to embed on your own website. Provides Webhooks.||Bad. Does not provide a Data Processing Agreement.||Good. User friendly interface and easy to get started.||3/4|
|Survey Monkey||Good. Has all possible question types and offers multiple-pages surveys.||Mixed. Easy to embed on your own website. Data can be automatically transferred to Google Sheets and others but requires third-party Software Zapier.||Bad. Still relies on EU-US Privacy Shield. Also, they process and use the answers they receive and sell insights they draw from them.||Good. User friendly interface and easy to get started.||2.5/4|
|Vue Formulate||Mixed. Provides you with most question types but not all. Multiple pages can be added with Vue.js directly.||Good. Since it provides only frontend, you can embed it easily on your website and choose your own way of transferring data.||Good. Since it provides only frontend, the server is up to you.||Bad. Requires frontend and backend skills to a certain degree.||2.5/4|
|VueJS Generators||Mixed. Provides you with most question types but not all. Multiple pages can be added with Vue.js directly.||Good. Since it provides only frontend, you can embed it easily on your website and choose your own way of transferring data.||Good. Since it provides only frontend, the server is up to you.||Bad. Requires frontend and backend skills to a certain degree.||2.5/4|
|Formik||Mixed. Provides you with most question types but not all. Multiple pages can be added with React directly.||Good. Since it provides only frontend, you can embed it easily on your website and choose your own way of transferring data.||Good. Since it provides only frontend, the server is up to you.||Bad. Requires frontend and backend skills to a certain degree.||2.5/4|