Energy Trading Forecast Platform Based on Real-time Metering Data

Quick Facts

CLIENT NAME

Forecast Platform Provider

INDUSTRY

Energy Trading

PROJECT NAME

Energy Trading Forecast Platform

SHORT DESCRIPTION OF THE SERVICES PROVIDED

Developing high throughput data pipelines for ingestion and enrichment of sensor data from multiple metering points across Europe. Delivering fully autonomous platform which serves real time decision-making statistics and forecasts to the Trading team. Ensuring the system reliability and performance by applying various types of automated tests.

TEAM SIZE

8 experts, including Business Analyst, Tech Lead, Solution Architect, DevOps/Performance Architect, Quality engineers and Data engineers

ENGAGEMENT PERIOD

September 2019 – ongoing

 

Technologies Involved

Apache Kafka, Apache Spark, Apache Camel, OpenTSDB, HBase, PostgreSQL, ELK stack for monitoring, Graphana, Jupyter, Java 8 for multiple custom services, Docker Swarm, Jenkins, Terraform

Business Challenge

The client is a holding company, which through its subsidiaries produces, procures, trades, and distributes electricity and gas. As the largest European municipal utility cooperation, the client opens up the potential of liberalized energy markets for municipal utilities along the entire value-added chain. Scale Focus was identified as their primary technology partner to integrate first-class processes in the software delivery pipeline, the result of which is an efficient, high-quality and resilient production system. Our engineering team is currently engaged in the development of their real-time decision-taking platform for intraday trading. The platform collects real time data points from multiple sensors of wind and solar renewable sources based in Europe, aggregates and correlates them with market prices and trading data and creates forecasts on top of statistical models developed by the best data scientists in the industry. The solution is perceived as critical given the fact that it enables the client to boost his profits, which on the other hand obliges our team to ensure its stability and fast implementation of additional features.

Solution

Key Features:

  • Technology partnership for building next generation IoT data collection and analytics platform with high value profitability.
  • Integration of CI/CD best practices to achieve fully automated provisioning and release life cycle in the Cloud
  • Using top of the edge solutions to achieve on-demand scalability
  • Complete data pipelines audit and performance assessment
  • Custom application and data flows development and quality assurance
  • Integration with multiple external data sources for real-time Market data, additional forecasts and general enrichment master data
  • Application management services, Business-as-Usual services
  • Real time data ingestion with high throughput and reliability

Goals

Key outcomes that our team aims for:

  • Direct communication with the business stakeholders in order to ensure smooth transition on all implementation objectives.
  • After detailed analysis of the requirements provided by the stakeholders to establish comprehensive architecture and design process .
  • Implementation of scalable and reliable system for processing thousands of data points per second
  • Code-quality analysis and checks, refactoring and formatting against the coding guidelines - automated by CI/CD
  • An established peer code review procedure, enforced by CI/CD and tools (Bitbucket / Jira)
  • A Software Development gap-analysis document: Outlined areas for improvement for better/consistent quality and control over the development, quality, infrastructure delivery.
  • Securing code repositories by applying regular automated backup procedure
  • Test-case management tool in use - for writing and executing test-cases, creating test-plans, etc.
  • Most critical business cases are covered by manual/automated test-cases
  • Bug-fixing / New feature development is performed in a timely manner and the system is properly maintained and supported. Reliable L2/L3 support service to ensure smooth continuity and knowledge transfer