Environmental Monitoring and IoT Data Analytics

The Case

The majority of businesses or government decision makers find the retrieval and analysis of climate and environmental data to help their decision-making processes extremely difficult. While the ability to access these data would be helpful, costly expert interpretation of the data is normally required to provide a meaningful answer to a business or public health questions. The University of Liverpool and Trendalyze are addressing these topics in the LEAF project, standing for Liverpool Environmental App Framework. LEAF processes the data to give user driven, expert solutions and products.

 

The aim of the project is to test the feasibility of a platform as a service solution, with LEAF placed in the “middle” of the frontend app and the back-end data collation/storage/analysis, built using open data, IoT data, visualization and analytics technologies, that will promote a market focused on answering specific business and public health questions related to the climate. This can be from solar power to health impacts.

The Solution

The University, Trendalyze and ScaleFocus engaged in a PoC with Science and Technology Facilities Council (STFC) to add Internet of Things (IoT) using multiple sensors from the STFC Hartree IoT campus and Liverpool University weather station blended with data from the Met Office. The software system makes apps that use environmental IoT/sensors and open data easier to develop. An app that works on a mobile phone, tablet or within a website has two ends: a front and a back.

An image with charts to describe anomaly chart in IoT data analytics

Figure 1: Anomaly chart

The front-end is the images and controls that the user sees and interacts with. The back-end is the workflow that communicates with a time series data analytics server. LEAF is the middle application core of the system, the concept and practical framework of joining the two ends and processing the data for users.

ScaleFocus got involved in the following areas of the technical solution:

  • Create Node-RED flows that process, reformat and store history and real-time data.
  • Create real-time data pipelines from given data sources, such as university weather station, forecast API (forecast.io).
  • Create simulated data streams to freeboard dashboards.
  • Create Spark Job Server using Scala that implements a search for sequence patterns, resulting in big performance boost over the Java code implementation of the same search.
  • Create graphs and visualisations using Zeppelin notebook (Scala + SparkSQL), displaying statistics over large datasets (averages, standard deviation, Pearson correlation, etc.).
  • Create and Android app.

 

Dashboard with a few metrics to descrbie IoT data analytics.

Full Technological Stack

 

Scala, Java, Android, JavaScript, SQL, Hadoop, Spark, Spark Job Sever, SparkSQL, Drill, Node-RED, Cloud environment (Microsoft Azure and IBM Softlayer), MQTT, Mosquitto Message Broker, IoT, Zeppelin notebook, Freeboard dashboard, Parquet

Achievements and Future

  • Creation of data pipelines to a cloud server, using the data at the University, being processed by Trendalyze’s tool
  • Creation of data processing framework for the University

The Partners

The University of Liverpool is a public university based in the city of Liverpool, England. Founded as a college in 1881, it gained its royal charter in 1903 with the ability to award degrees. It comprises three faculties organised into 35 departments and schools. It is a founding member of the Russell Group of research-intensive universities.

Trendalyze is a company focused on innovation in analytics – they build the first motif discovery platform to help professionals understand patterns in data. Their solution enables to unlock the value of time patterns and empowers users to reveal root causes of events, share their discoveries, and monitor for patterns to improve outcomes in their industries.

The University of Liverpool and Trendalyze are cooperating in various initiatives, digging into cutting-edge areas for IoT and Analytics, mainly bridging the discoveries of Trendalyze, with the huge datasets that the university has access to, in the context of processing, statistics’ and trends’ visualization.