Updated 06 April 2021.
Copyright: European Union Public License, version 1.2 (EUPL-1.2).
We designed a protocol for testing applications so as to be transparent and credible for our sponsors and readers and have consistency across our reviews.
The aim of the review is not to judge or recommend, but to use the app and thoroughly review all of its features, so that end users can decide according to their situations at a certain point in time and space which apps to use.
We will thoroughly review the app, from a security and usability perspective and publish an article. We test the app from a business standpoint.
- Does it add value to your organisation?
- Does it increase reach to your audience?
- Is it friction-less for users to adopt?
1- Lists of features.
- We have a generic lists for:
- Communication apps,
- Website and blogs,
- Operating systems,
- Communications apps means sending text (email or chat), image, video, VoIP meeting, video conferencing, file sharing.
- We go through the entire list as for a check list, even if the feature doesn’t seem relevant. That way it is easy for our readers toput reviews side by side and compare apps.
- We add features in no particular order, as we come across the need to integrate them into our list.
- The feature list is here to help us not overlook some features. As a result, some are antinomic, for example between privacy and usability. Some are irrelevant.
To ensure an unbiased and thorough review all apps are tested:
- In real time, i.e. we use it on real projects.
- Among different team members located in different countries. If possible in countries where governments are trying to block internet, so as to see how the application can handle these issues.
- If possible we put a certified ethical hacker (white hat) into the team, so as to look into the code or try to intercept communications and break into the device.
- With different devices and operating systems. For desktop testing, we use Linux and Windows at the minimum, if possible macOS. On portable devices, we try the application on Android and iOS at the minimum, if possible we use AOSP and Linux. We also test the alternative app stores to see if the application is available outside of Google Play or Apple Store, for example we test if the application is available on APK _Pure and F_Droid.
- For a minimum of two weeks, four on average.
- The article is peer-reviewed by other team members then sent to the app developers for final review.
3- Milestones, Goals.
In order to make it easier for contributors to understand how we are going to use the funds, we’ve set milestones, called “goals” on the Open Collective platform. The standard list of milestones goes as this:
- 1-Threat Model: Define a threat model scenario to benchmark the app against.
- 2-Specifications sheet: Define the needs of end users according to threat model. Define specifications sheet with a list of features to be reviewed.
- 3-Gather team: Based on the threat model scenario, gather a team of skilled end users in the field to review the app. Find them in different countries in order to assess the app on an international setting.
- 4-Comprehensive review of the app week 1: Review each feature of the app. Draft the article in English.
- 5-Comprehensive review of the app week 2: Review each feature of the app. Draft the article in English.
- 6-Comprehensive review of the app week 3: Review each feature of the app. Draft the article in English.
- 7-Comprehensive review of the app week 4: Review each feature of the app. Draft the article in English.
- 8-Ethical hacker: Hire Ethical hacker (aka: white-hat) to test the vulnerabilities of the app.
- 9-Finalise article: Based on the report from the ethical hacker, finalise the article and submit it to App developers for comments, update article accordingly.
- 10-Translations: Translate the article in various languages depending on the countries we cover at that time. Example: French, Estonian, Spanish, Portuguese. Share article with sponsors, in all languages.
- 11-Publishing: Share article with sponsors, in all languages, under the Copyright European Union Public License, version 1.2 (EUPL-1.2). 3 months later, publish the article on our websites.