I want to work with data
by ensuring data quality

Data reliability (or replicability) is about whether you can get the same results when you repeat an experiment or observation. For example, suppose that you see a bird and identify that bird as a robin. At the same time, someone else sees the same bird and comes to the same conclusion. The data is reliable because multiple observations have given the same result.

The participation of many observers in your Citizen Observatory with different degrees of expertise can be compensated by taking advantage of expert contributors who can review the observations of others. When citizens use inexpensive or DIY sensors connected to smartphones, there are also concerns about data quality and stability of the measurements. In the sensor.community initiative, citizens are encouraged to regularly collect data close to official stations as a way to estimate the bias and degradation of low-cost sensors and to ensure compatibility with official sources.

A lack of citizen expertise can also be mitigated by providing comprehensive and focused training for new participants in the Citizen Observatory in order to ensure they know how to collect valid observations. This training can be done in the form of face-to-face workshops, online video tutorials or courses, and can be complemented by extensive training materials on the website of the Citizen Observatory.

Data quality at the observation level can also be improved by providing well-designed data collection tools that minimise the chances of data collection errors. For qualitative values, multiple-choice selectors can be provided instead of free text inputs. For quantitative values measured by soil and air temperature sensors, values outside of a predefined range can be considered wrong and eliminated before allowing values into the system storage.

Example from the LandSense project

LandSense also uses a quality assurance system to make sure data collected for land cover detection, agricultural monitoring and habitat monitoring campaigns are ‘clean’. First, the system checks for overlaps in areas drawn by users and flags them so that users can correct them. Next, the system looks for problems with photographs. Many citizen science projects have mobile apps that ask citizens to take photographs as part of the data collection process, but this can create issues of personal privacy. For example, faces and license plates are automatically blurred out to comply with the EU General Data Protection Regulation (GDPR). Photographs are also checked to make sure they are not too dark or blurry. Next, the service checks for position accuracy, using mobile phone GPS to make sure an observation is geographically accurate. It also checks against reference data from a ‘gold standard’ data set produced by professional scientists to make sure there is an overall agreement and that nothing is omitted or entered incorrectly. Finally, the service compares answers from the same location given by multiple contributors to provide a level of confidence in the data.

Example of a photograph from the CityOases mobile app with blurring from the LandSense quality assurance service applied to faces

Reliability in Machine Learning

Common examples of non-reliable data are observations provided by uncalibrated sensors (instrument biases) or human misinterpretations, but reliability problems can also emerge in artificial intelligence. In the Scent project, citizens are guided to areas where there is a need for environmental information. In those locations, they collect images of land cover/land use. Images can be submitted to the ScentIntelligence Engine (SIE); a tool that uses machine learning to automatically detect land cover types and objects in an image according to Scent’s taxonomy. The system works by assigning a score to each annotation tag. If the score is high enough, the annotation is considered valid. If not, the image is redirected to Scent Collaborate, where citizen scientists manually annotate the images.